From christos at zoulas.com Sat Feb 8 13:40:45 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:55 2005 Subject: file-3.40 is now available Message-ID: <20030208184045.A53FF7E3A@beowulf.gw.com> from ftp.astron.com:/pub/file/file-3.40.tar.gz Enjoy, christos - detect inttypes.h too (Dave Love ) - eliminate unsigned char warnings (Petter Reinholdtsen ) - better elf PT_NOTE handling (Nalin Dahyabhai ) - add options to format the output differently - much more magic. From ian at darwinsys.com Mon Feb 24 19:42:08 2003 From: ian at darwinsys.com (Ian Darwin) Date: Sat Mar 5 00:36:55 2005 Subject: Fwd: security vulnerability in File Message-ID: <200302241942.08662.ian@darwinsys.com> I've received what looks like a legitimate report of a security buffer overflow-type problem in file up to 3.39. Can somebody who is more familiar with the ELF module please check into this fairly quickly and get back do David Endler before he goes public with this? Thanks Ian ---------- Forwarded Message ---------- Subject: security vulnerability in File Date: February 24, 2003 04:07 pm From: David Endler To: "'mail_contact@darwinsys.com'" In accordance with iDEFENSE's security vulnerability disclosure policy (www.idefense.com/disclosure.html), we wanted to bring a security issue to your attention. Please acknowledge receipt of this information so we can work toward a public disclosure date that is convenient to you in building a fix. David Endler, CISSP Director, Technical Intelligence iDEFENSE, Inc. 14151 Newbrook Drive Suite 100 Chantilly, VA 20151 voice: 703-344-2632 fax: 703-961-1071 dendler@idefense.com www.idefense.com Ian Darwin's File command contains a buffer overflow vulnerability that can be leveraged by an attacker to execute arbitrary code under the privileges of another user. The crux of the problem lies in the following call to doshn() from tryelf() on line 587 in readelf.c: doshn(class, swap, fd, getu32(swap, elfhdr.e_shoff), getu16(swap, elfhdr.e_shnum), getu16(swap, elfhdr.e_shentsize)); The final argument to doshn() 'elfhdr.e_shentsize' is later used in a call to read() as can be see here on line 133 in readelf.c: if (read(fd, sh_addr, size) == -1) The call to read() will copy 'size' bytes into the variable 'sh_addr' which is defined on line 92 in readelf.c: #define sh_addr (class == ELFCLASS32 \ ? (void *) &sh32 \ : (void *) &sh64) The storage buffer used in the call to read() is of size 0x20 (32) bytes, by supplying a 'size' of 0x28 (40) a stack overflow occurs overwriting the stored frame pointer (EBP) and instruction pointer (EIP) thereby providing the attacker with CPU control and the ability to execute arbitrary code. Analysis: (iDEFENSE US) A user who can successfully convince another user to examine a specially constructed exploit file with the File command can execute arbitrary code under the privileges of that user. File is an application that utilizes a magic file (typically located in /usr/share/magic) to classify arbitrary files. The latest version of File is available for download from: ftp://ftp.astron.com/pub/file. Detection: iDEFENSE has successfully exploited File version 3.37 and 3.39. It is suspected that all versions up to an including the latest version (3.39) are vulnerable. Exploit: The following is a sample walkthrough of a successful exploitation. The attacker must initially generate a file that is specially structured to trigger a buffer overflow in the File command: $ ./mkfile_expl -C /tmp/suid -F /tmp/exploit -O "ASCII text" -R /bin/bash -p 1 Local /usr/bin/file upto v3.39 exploit by anonymous Using PRESET: 1 [Linux file <= 3.38 ] Using FILENAME: /tmp/exploit Using REAL_SHELL: /bin/bash Using CREATED_SHELL: /tmp/suid Using OUTPUT: ASCII text Using RET_ADDR: 0xbfffc3f0 Using NOP_COUNT: 6000 Exploit created -> /tmp/exploit Time to wait till somebody starts /usr/bin/file /tmp/exploit Once the tainted file has been generated the attacker must wait for or coerce another user to examine the file with the File command. # ls -l exploit -rwxr-xr-x 1 farmer farmer 6406 Jan 11 22:07 exploit # file exploit /tmp/exploit: ASCII text The File command reports that the examined file is "ASCII text" as the attacker specified in the creation of the exploit file. At this point if the attack was a success the original attack file (exploit) has been erased and a set user id shell has been created: # ls -l exploit ls: exploit: No such file or directory $ ls -l suid -rwsr-sr-x 1 root root 541096 Jan 11 22:07 suid The proof of concept exploit code has been attached to this message. Workaround: In order to increase the chances of exploitation an attacker inserts many NO Operation (NOP) instructions thereby generated a "sled" to create a greater range of addresses that will lead to the execution of arbitrary code. A user who suspects a certain file may indeed be tainted can check for the existence of a large number of consecutive NOP instructions (0x90) using the hexdump command. The following is an excerpt of a hexdump of a sample exploit file: 0000ec0 9090 9090 9090 9090 9090 9090 9090 9090 0000ed0 9090 9090 9090 9090 9090 9090 9090 9090 0000ee0 9090 9090 9090 9090 9090 9090 9090 9090 0000ef0 9090 9090 9090 9090 9090 9090 9090 9090 0000f00 9090 9090 9090 9090 9090 9090 9090 9090 0000f10 9090 9090 9090 9090 9090 9090 9090 9090 ------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: mkfile_expl.c Type: application/octet-stream Size: 23843 bytes Desc: not available Url : http://mx.gw.com/pipermail/file/attachments/20030224/28754d2e/mkfile_expl.obj From christos at zoulas.com Tue Feb 25 07:59:02 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:55 2005 Subject: Fwd: security vulnerability in File In-Reply-To: <200302241942.08662.ian@darwinsys.com> from Ian Darwin (Feb 24, 7:42pm) Message-ID: <20030225125902.391BB7E4C@beowulf.gw.com> On Feb 24, 7:42pm, ian@darwinsys.com (Ian Darwin) wrote: -- Subject: Fwd: security vulnerability in File | I've received what looks like a legitimate report of a security | buffer overflow-type problem in file up to 3.39. Can somebody | who is more familiar with the ELF module please check into this | fairly quickly and get back do David Endler before he goes public | with this? | | Thanks | Ian This is correct, here's a patch and it will be fixed in the next version of file. I am not sure the code audit did not find a memory allocation bug in softmagic.c... christos RCS file: /src/pub/file/readelf.c,v retrieving revision 1.23 diff -u -u -r1.23 readelf.c --- readelf.c 8 Feb 2003 18:33:53 -0000 1.23 +++ readelf.c 25 Feb 2003 12:54:13 -0000 @@ -133,7 +133,7 @@ error("lseek failed (%s).\n", strerror(errno)); for ( ; num; num--) { - if (read(fd, sh_addr, size) == -1) + if (read(fd, sh_addr, sizeof(*sh_addr)) == -1) error("read failed (%s).\n", strerror(errno)); if (shs_type == SHT_SYMTAB /* || shs_type == SHT_DYNSYM */) { (void) printf (", not stripped"); From christos at zoulas.com Tue Feb 25 10:32:37 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:55 2005 Subject: Fwd: security vulnerability in File In-Reply-To: <200302241942.08662.ian@darwinsys.com> from Ian Darwin (Feb 24, 7:42pm) Message-ID: <20030225153237.3135D7E4B@beowulf.gw.com> On Feb 24, 7:42pm, ian@darwinsys.com (Ian Darwin) wrote: -- Subject: Fwd: security vulnerability in File | I've received what looks like a legitimate report of a security | buffer overflow-type problem in file up to 3.39. Can somebody | who is more familiar with the ELF module please check into this | fairly quickly and get back do David Endler before he goes public | with this? | | Thanks | Ian Actually the program headers can be taken advantage to the same way. I have a more complete patch. christos From christos at zoulas.com Thu Feb 27 10:50:02 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:55 2005 Subject: Fwd: security vulnerability in File In-Reply-To: <200302271513.JAA23277@juno.eecs.tulane.edu> from David Endler (Feb 27, 9:13am) Message-ID: <20030227155002.D66717E3A@beowulf.gw.com> On Feb 27, 9:13am, endler@eecs.tulane.edu (David Endler) wrote: -- Subject: Re: Fwd: security vulnerability in File | > Actually, the program headers can be taken advantage of the same way | > I have a more complete patch | | Christos, Ian, etc al, | | Can you send the final patch along so we can plan a public advisory and | inform the OS vendors? Thanks, Here's my final patch, including the other memory overflow in softmagic I mentioned. Regards, christos --- readelf.c 8 Feb 2003 18:33:53 -0000 1.23 +++ readelf.c 25 Feb 2003 15:30:00 -0000 1.26 @@ -92,12 +92,18 @@ #define sh_addr (class == ELFCLASS32 \ ? (void *) &sh32 \ : (void *) &sh64) +#define sh_size (class == ELFCLASS32 \ + ? sizeof sh32 \ + : sizeof sh64) #define shs_type (class == ELFCLASS32 \ ? getu32(swap, sh32.sh_type) \ : getu32(swap, sh64.sh_type)) #define ph_addr (class == ELFCLASS32 \ ? (void *) &ph32 \ : (void *) &ph64) +#define ph_size (class == ELFCLASS32 \ + ? sizeof ph32 \ + : sizeof ph64) #define ph_type (class == ELFCLASS32 \ ? getu32(swap, ph32.p_type) \ : getu32(swap, ph64.p_type)) @@ -129,11 +135,14 @@ Elf32_Shdr sh32; Elf64_Shdr sh64; + if (size != sh_size) + error("corrupted section header size.\n"); + if (lseek(fd, off, SEEK_SET) == -1) error("lseek failed (%s).\n", strerror(errno)); for ( ; num; num--) { - if (read(fd, sh_addr, size) == -1) + if (read(fd, sh_addr, sh_size) == -1) error("read failed (%s).\n", strerror(errno)); if (shs_type == SHT_SYMTAB /* || shs_type == SHT_DYNSYM */) { (void) printf (", not stripped"); @@ -162,11 +171,13 @@ size_t offset, nameoffset; off_t savedoffset; + if (size != ph_size) + error("corrupted program header size.\n"); if (lseek(fd, off, SEEK_SET) == -1) error("lseek failed (%s).\n", strerror(errno)); for ( ; num; num--) { - if (read(fd, ph_addr, size) == -1) + if (read(fd, ph_addr, ph_size) == -1) error("read failed (%s).\n", strerror(errno)); if ((savedoffset = lseek(fd, 0, SEEK_CUR)) == -1) error("lseek failed (%s).\n", strerror(errno)); @@ -360,13 +371,15 @@ int bufsize; int os_style = -1; + if (size != ph_size) + error("corrupted program header size.\n"); /* * Loop through all the program headers. */ for ( ; num; num--) { if (lseek(fd, off, SEEK_SET) == -1) error("lseek failed (%s).\n", strerror(errno)); - if (read(fd, ph_addr, size) == -1) + if (read(fd, ph_addr, ph_size) == -1) error("read failed (%s).\n", strerror(errno)); off += size; if (ph_type != PT_NOTE) --- softmagic.c 8 Feb 2003 18:33:53 -0000 1.52 +++ softmagic.c 25 Feb 2003 13:04:32 -0000 1.54 @@ -105,7 +105,8 @@ int firstline = 1; /* a flag to print X\n X\n- X */ if (tmpoff == NULL) - if ((tmpoff = (int32_t *) malloc(tmplen = 20)) == NULL) + if ((tmpoff = (int32_t *) malloc( + (tmplen = 20) * sizeof(*tmpoff))) == NULL) error("out of memory\n"); for (magindex = 0; magindex < nmagic; magindex++) { @@ -137,7 +138,7 @@ /* and any continuations that match */ if (++cont_level >= tmplen) if ((tmpoff = (int32_t *) realloc(tmpoff, - tmplen += 20)) == NULL) + (tmplen += 20) * sizeof(*tmpoff))) == NULL) error("out of memory\n"); while (magic[magindex+1].cont_level != 0 && ++magindex < nmagic) { @@ -184,7 +185,8 @@ if (++cont_level >= tmplen) if ((tmpoff = (int32_t *) realloc(tmpoff, - tmplen += 20)) == NULL) + (tmplen += 20) + * sizeof(*tmpoff))) == NULL) error("out of memory\n"); } if (magic[magindex].flag & OFFADD) { From christos at zoulas.com Thu Feb 27 16:00:03 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:55 2005 Subject: file-3.41 is now available Message-ID: <20030227210003.818737E4B@beowulf.gw.com> This was really not a planned release, but might as well address the stack overflow issues that idefense has found and the memory allocation problems discovered by redhat. * - fix memory allocation problem (Jeff Johnson) * - fix stack overflow corruption (David Endler) * - fixes from NetBSD source (Antti Kantee) * - magic fixes The next release of file is going to take a bit because I am working on libmagic, a small api that lets people use file(1) functionality programatically. Regards, christos From christos at zoulas.com Thu Feb 27 16:00:51 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:55 2005 Subject: file-3.41 [part 2] Message-ID: <20030227210051.B06F37E4C@beowulf.gw.com> For the new-comers, this is available from: ftp.astron.com:/pub/file/file-3.41.tar.gz christos From soren at wheel.dk Thu Feb 27 17:35:57 2003 From: soren at wheel.dk (Soren S. Jorvang) Date: Sat Mar 5 00:36:55 2005 Subject: file/magdir/cvs vs. CVS Message-ID: <20030227223557.GA6079@gnyf.wheel.dk> Having the file cvs in magdir causes problems when using it under CVS on case insensitive filesystems such as MacOS (X) HFS. I suggest that the file be renamed to remove this conflict. -- Soren From christos at zoulas.com Sat Mar 1 20:01:07 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: Fwd: security vulnerability in File In-Reply-To: <200302271513.JAA23277@juno.eecs.tulane.edu> from David Endler (Feb 27, 9:13am) Message-ID: <20030302010107.E6B5C7E3A@beowulf.gw.com> On Feb 27, 9:13am, endler@eecs.tulane.edu (David Endler) wrote: -- Subject: Re: Fwd: security vulnerability in File | > Actually, the program headers can be taken advantage of the same way | > I have a more complete patch | | Christos, Ian, etc al, | | Can you send the final patch along so we can plan a public advisory and | inform the OS vendors? Thanks, | I did send the final patch... christos From christos at zoulas.com Sat Mar 1 20:01:27 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: file/magdir/cvs vs. CVS In-Reply-To: <20030227223557.GA6079@gnyf.wheel.dk> from "Soren S. Jorvang" (Feb 27, 11:35pm) Message-ID: <20030302010127.371B97E4D@beowulf.gw.com> On Feb 27, 11:35pm, soren@wheel.dk ("Soren S. Jorvang") wrote: -- Subject: file/magdir/cvs vs. CVS thanks, I'll do that. christos | Having the file cvs in magdir causes problems when using it under | CVS on case insensitive filesystems such as MacOS (X) HFS. | | I suggest that the file be renamed to remove this conflict. | | | -- | Soren | _______________________________________________ | File mailing list | File@mx.gw.com | http://mx.gw.com/mailman/listinfo/file -- End of excerpt from "Soren S. Jorvang" From ian at darwinsys.com Sat Mar 1 18:57:39 2003 From: ian at darwinsys.com (Ian F. Darwin) Date: Sat Mar 5 00:36:56 2005 Subject: libfile and MIME integration Message-ID: <200303011139.28349.ian@darwinsys.com> On February 27, 2003 04:00 pm, Christos Zoulas wrote: > The next release of file is going to take a bit because I am working > on libmagic, a small api that lets people use file(1) functionality > programatically. I thought this would be coming soon (I was about to suggest it otherwise). The GNU project mkhybrid did this and called it libfile; I presume you've looked at their work and are using the same or similar API? Anyway, my suggestion is on how to do a better job of merging mime types and textual file types, so that only a single data file needs to be maintained. The present scheme with types in "magic" also being present in magic2mime with their MIME correspondents, is not maintainable and violates the "information in one place only" pattern. I thought of something like this: 0 long 0x20 { abeco image data image/abeco } That is, a Struct-like syntax that includes both the standard file response as now, which can be continued etc., as at present, and a single line for the MIME type. This has the advantage that you can maintain backwards compatible files (either the file string field contains a { or it doesn't), and in libmagic we'll have both file types and mime types. The API should also be used in Apache mod_file, to get rid of the "code re-use via the copy-and-paste design pattern" that is currently deployed there. Ian From christos at zoulas.com Sun Mar 2 13:20:29 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: libfile and MIME integration In-Reply-To: <200303011139.28349.ian@darwinsys.com> from "Ian F. Darwin" (Mar 1, 6:57pm) Message-ID: <20030302182029.CA3067E4C@beowulf.gw.com> On Mar 1, 6:57pm, ian@darwinsys.com ("Ian F. Darwin") wrote: -- Subject: libfile and MIME integration | On February 27, 2003 04:00 pm, Christos Zoulas wrote: | > The next release of file is going to take a bit because I am working | > on libmagic, a small api that lets people use file(1) functionality | > programatically. | | I thought this would be coming soon (I was about to suggest it otherwise). | The GNU project mkhybrid did this and called it libfile; I presume you've | looked at their work and are using the same or similar API? I did not know about that, thanks for pointing it out to me. I will take a look at it. | Anyway, my suggestion is on how to do a better job of merging mime | types and textual file types, so that only a single data file needs to be | maintained. The present scheme with types in "magic" also being present in | magic2mime with their MIME correspondents, is not maintainable and violates | the "information in one place only" pattern. I thought of something like | this: | | 0 long 0x20 { | abeco image data | image/abeco | } | | That is, a Struct-like syntax that includes both the standard file response as | now, which can be continued etc., as at present, and a single line for the | MIME type. | | This has the advantage that you can maintain backwards compatible files | (either the file string field contains a { or it doesn't), and in libmagic | we'll have both file types and mime types. If we are going to make changes to the syntax, then we should also address how to handle conflicts and priority of matches/multiple matches. Maybe for the next version after the library release. | The API should also be used in Apache mod_file, to get rid of the "code re-use | via the copy-and-paste design pattern" that is currently deployed there. | I totally agree, christos From christos at zoulas.com Tue Mar 18 15:40:22 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: libfile and MIME integration In-Reply-To: <200303011139.28349.ian@darwinsys.com> from "Ian F. Darwin" (Mar 1, 6:57pm) Message-ID: <20030318204022.637E57E4D@beowulf.gw.com> On Mar 1, 6:57pm, ian@darwinsys.com ("Ian F. Darwin") wrote: -- Subject: libfile and MIME integration | I thought this would be coming soon (I was about to suggest it otherwise). | The GNU project mkhybrid did this and called it libfile; I presume you've | looked at their work and are using the same or similar API? FYI: I looked at they work, and I have discarded their api. It does not have a namespace (consistent naming of functions), does not report errors properly, and has no way to pass in flags. In other words, it is nothing more than a single afternoon's hack. I am still working on integrating patches from libmagic that was sent to me via e-mail. christos From christos at zoulas.com Mon Mar 24 10:19:49 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.00 is now available Message-ID: <20030324151949.3868D7E3A@beowulf.gw.com> File 4.00 is now available. This is a .00 release, i.e. expect to find problems with it. On the other hand, I encourage you to try it out, as this is the first version of file based on libmagic(3). The initial breakup of the code and the basic API idea comes from M\xe5ns Rullg\xe5rd, with a lot of re-writing and cleaning up by me. Other changes: apptype.c for __EMX__ (John Poltorak) better handling of elf notes (Jason Thorpe) Please feel free to comment on the api, as well as build problems, warnings etc. christos From obrien at NUXI.com Mon Mar 24 14:14:24 2003 From: obrien at NUXI.com (David O'Brien) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.00 is now available In-Reply-To: <20030324151949.3868D7E3A@beowulf.gw.com> References: <20030324151949.3868D7E3A@beowulf.gw.com> Message-ID: <20030324191424.GA55005@dragon.nuxi.com> On Mon, Mar 24, 2003 at 10:19:49AM -0500, Christos Zoulas wrote: > > File 4.00 is now available. Hi Christos, I looked for it at the usual place, but didn't find it. Do you have a URL for the new release? thanks, -- -- David (obrien@NUXI.com) From kim at tac.nyc.ny.us Mon Mar 24 18:40:43 2003 From: kim at tac.nyc.ny.us (Kimmo Suominen) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.00 is now available In-Reply-To: <20030324191424.GA55005@dragon.nuxi.com> from "David O'Brien" on Mon, 24 Mar 2003 11:14:24 -0800 References: <20030324151949.3868D7E3A@beowulf.gw.com> <20030324191424.GA55005@dragon.nuxi.com> Message-ID: <20030324234043.5EA407E4B@beowulf.gw.com> Try ftp://ftp.astron.com/pub/file/ This is what gets mirrored daily to ftp://ftp.gw.com/pub/unix/file/ ftp://ftp.funet.fi/pub/unix/tools/file/ Regards, + Kim | From: "David O'Brien" | Date: Mon, 24 Mar 2003 11:14:24 -0800 | | On Mon, Mar 24, 2003 at 10:19:49AM -0500, Christos Zoulas wrote: | > | > File 4.00 is now available. | | Hi Christos, | | I looked for it at the usual place, but didn't find it. Do you have a | URL for the new release? | | thanks, | -- | -- David (obrien@NUXI.com) From christos at zoulas.com Tue Mar 25 07:21:05 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.00 is now available In-Reply-To: <20030324191424.GA55005@dragon.nuxi.com> from "David O'Brien" (Mar 24, 11:14am) Message-ID: <20030325122105.BE8417E4B@beowulf.gw.com> On Mar 24, 11:14am, obrien@NUXI.com ("David O'Brien") wrote: -- Subject: Re: file-4.00 is now available Hi David, ftp.astron.com:/pub/file/file-4.00.tar.gz Enjoy, christos | > File 4.00 is now available. | | Hi Christos, | | I looked for it at the usual place, but didn't find it. Do you have a | URL for the new release? | | thanks, | -- | -- David (obrien@NUXI.com) | _______________________________________________ | File mailing list | File@mx.gw.com | http://mx.gw.com/mailman/listinfo/file -- End of excerpt from "David O'Brien" From obrien at NUXI.com Tue Mar 25 12:35:42 2003 From: obrien at NUXI.com (David O'Brien) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.00 is now available In-Reply-To: <20030325122105.BE8417E4B@beowulf.gw.com> References: <20030324191424.GA55005@dragon.nuxi.com> <20030325122105.BE8417E4B@beowulf.gw.com> Message-ID: <20030325173542.GA58074@dragon.nuxi.com> On Tue, Mar 25, 2003 at 07:21:05AM -0500, Christos Zoulas wrote: > On Mar 24, 11:14am, obrien@NUXI.com ("David O'Brien") wrote: > -- Subject: Re: file-4.00 is now available > > Hi David, > > ftp.astron.com:/pub/file/file-4.00.tar.gz Opps, sorry. I had a brain fart and looked at ftp.gw.com, which didn't have it yet. From christos at zoulas.com Wed Mar 26 11:42:11 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.01 is now available Message-ID: <20030326164211.596557E57@beowulf.gw.com> >From: ftp.astron.com:/pub/file Thank you everyone for your feedback and bug reports. Things fixed: * Pass lint * make NULL in magic_file mean stdin * Fix "-" argument to file to pass NULL to magic_file * avoid pointer casts by using memcpy * rename magic_buf -> magic_buffer * keep only the first error * manual page: new sentence, new line * fix typo in api function (magic_buf -> magic_buffer) christos From ian at darwinsys.com Thu Mar 20 11:33:53 2003 From: ian at darwinsys.com (Ian F. Darwin) Date: Sat Mar 5 00:36:56 2005 Subject: libfile and MIME integration In-Reply-To: <20030318204022.637E57E4D@beowulf.gw.com> References: <20030318204022.637E57E4D@beowulf.gw.com> Message-ID: <200303201133.53245.ian@darwinsys.com> > I looked at [mkhybrid libfile], and I have discarded their api. It does not > have a namespace (consistent naming of functions), does not report > errors properly, and has no way to pass in flags. In other words, > it is nothing more than a single afternoon's hack. I am still > working on integrating patches from libmagic that was sent to me > via e-mail. Well, thanks for checking. I have a new file "binmagic" which calls is_tar and does a few other things that need more work than can easily be done in the magic file at present. Is it OK if I commit this file but not yet commit the changes to main to hook it in, so people can get a look at it? I don't know how you normally coordinate CVS changes and I don't want to interfere with the work you're doing. The "binmagic" stuff is, e.g., is_tar, getting the sizes of various binary image files (jpeg), reading ISO9660 CD's and printing various fields from the header neatly, etc. Thanks Ian From christos at zoulas.com Sat Mar 29 15:46:46 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: libfile and MIME integration In-Reply-To: <200303201133.53245.ian@darwinsys.com> from "Ian F. Darwin" (Mar 20, 11:33am) Message-ID: <20030329204646.CE9437E3A@beowulf.gw.com> On Mar 20, 11:33am, ian@darwinsys.com ("Ian F. Darwin") wrote: -- Subject: Re: libfile and MIME integration Sure, check it in... The hooks are now in funcs.c... It should be pretty easy to add. christos | I have a new file "binmagic" which calls is_tar and does a few other | things that need more work than can easily be done in the magic file | at present. Is it OK if I commit this file but not yet commit the changes | to main to hook it in, so people can get a look at it? I don't know how | you normally coordinate CVS changes and I don't want to interfere | with the work you're doing. | | The "binmagic" stuff is, e.g., is_tar, getting the sizes of various | binary image files (jpeg), reading ISO9660 CD's and printing | various fields from the header neatly, etc. | | Thanks | Ian | | _______________________________________________ | File mailing list | File@mx.gw.com | http://mx.gw.com/mailman/listinfo/file -- End of excerpt from "Ian F. Darwin" From christos at zoulas.com Wed Apr 2 14:19:32 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: file-4.02 is now available Message-ID: <20030402191932.26BA57E4E@beowulf.gw.com> Hi, This version of file fixes all known problems, and I consider it stable enough to replace 3.41. In particular, the file compiled format has been verified to work with both 32 and 64 big and little endian machines. Thanks to everyone for reporting problems. Here's what 4.02 fixes: 2003-04-02 13:50 Christos Zoulas * Magic additions (Alex Ott) * Fix bug that broke VPATH compilation (Peter Breitenlohner) 2003-03-28 16:03 Christos Zoulas * remove packed attribute from magic struct. * make the magic struct properly aligned. * bump version number of compiled files to 2. 2003-03-27 13:10 Christos Zoulas * separate tar detection and run it before softmagic. * fix reversed symlink test. * fix version printing. * make separator a string instead of a char. * update manual page and sort options. This is available as always from: ftp.astron.com:/pub/file/file-4.02.tar.gz Enjoy, christos From ian at darwinsys.com Fri May 16 22:20:16 2003 From: ian at darwinsys.com (Ian Darwin) Date: Sat Mar 5 00:36:56 2005 Subject: [ian@darwinsys.com: thoughts on libmagic] Message-ID: <20030516222016.A14936@www.darwinsys.com> Two issues: 1) What would you think about splitting "src" into two directories, one for libmagic and the other for file? It turns out easier for support on BSD systems to have each in its own directory, so you can use and 2) I'm thinking out loud here, but there ought to be a more general structure to the interaction with libmagic, something like magic_open_file(const char *filename); which would either set state or, preferably, return an opaque struct, and in either case could be followed up with macros/functions. My general model here is stat(2). Things we have wanted to know in an application include the following: boolean is_binary() / is_text() Lots of programs want to know this, e.g., lpr to know if it's safe to print a file. char *get_type() to get the file type (what file normally prints) char *get_mime_type() to get the mime type. A web server for example might want to print a directory listing and would use get_type() in the printed listing but would use get_mime_type() to actually send the file. The notion as now that this must be set at magic_open time, or twiddled using flags, strikes me as improvable. What do the rest of you think? Ian From christos at zoulas.com Sat May 24 14:18:20 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: file 4.03 is now available Message-ID: <20030524181820.E4C6A7E4D@beowulf.gw.com> Hello, file-4.03 is now available from ftp.astron.com:/pub/file/file-4.03.tar.gz This version contains numerous fixes: * documentation fixes from Michael Piefel * magic fixes (various) * revert basename magic in .mgc name determination * buffer protection in uncompress, signness issues, close files Maciej W. Rozycki Hello, file-4.04.tar.gz is available from ftp.astron.com:/pub/file. This version features: - many magic fixes - memory leak fixes - portability fixes See the ChangeLog for more information. Enjoy, christos From brettf at deepfile.com Tue Oct 7 15:57:26 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:56 2005 Subject: libmagic python bindings Message-ID: <1690.192.168.123.158.1065556646.squirrel@staff.deepfile.com> Greetings, I have written a python wrapper around libmagic. I'm wondering if the maintainers would like to add it to the source distribution. I'm currently building it by hand (using distutils) after going through the configure, make, make install routine for 'file' but it might be nice to have the python module build process integrated with the regular build for 'file'. Perhaps something like ./configure --with-python. Unfortunately, I've never used autoconf/automake so I have no idea how to do this. I've posted the .c file for review. Hit me back if there's interest and I'll post the .h and distutils build file. Regards, brett -------------- next part -------------- A non-text attachment was scrubbed... Name: py_magic.c Type: application/octet-stream Size: 9088 bytes Desc: not available Url : http://mx.gw.com/pipermail/file/attachments/20031007/09c806d4/py_magic.obj From christos at zoulas.com Tue Oct 7 19:42:09 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:56 2005 Subject: libmagic python bindings In-Reply-To: <1690.192.168.123.158.1065556646.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 7, 2:57pm) Message-ID: <20031007234209.53FE17E05@beowulf.gw.com> On Oct 7, 2:57pm, brettf@deepfile.com ("Brett Funderburg") wrote: -- Subject: libmagic python bindings Yes, definitely there is interest. At least as a first step, we can include the source and then we can visit the autoconf stuff later. I am an autoconf dummy too (and I like to stay that way). christos | Greetings, | | I have written a python wrapper around libmagic. I'm wondering if the | maintainers would like to add it to the source distribution. I'm currently | building it by hand (using distutils) after going through the configure, | make, make install routine for 'file' but it might be nice to have the | python module build process integrated with the regular build for 'file'. | Perhaps something like ./configure --with-python. Unfortunately, I've | never used autoconf/automake so I have no idea how to do this. | | I've posted the .c file for review. Hit me back if there's interest and | I'll post the .h and distutils build file. | | Regards, | | brett | ------=_20031007145726_30296 | Content-Type: application/octet-stream; name="py_magic.c" | Content-Transfer-Encoding: base64 | Content-Disposition: attachment; filename="py_magic.c" | | LyoKICAgUHl0aG9uIHdyYXBwZXJzIGZvciBtYWdpYyBmdW5jdGlvbnMuCgogICBDb3B5cmlnaHQg | KEMpIEJyZXR0IEZ1bmRlcmJ1cmcsIERlZXBmaWxlIENvcnAuIEF1c3RpbiwgVFgsIFVTIDIwMDMK | CiAgIFRoaXMgcHJvZ3JhbSBpcyBmcmVlIHNvZnR3YXJlOyB5b3UgY2FuIHJlZGlzdHJpYnV0ZSBp | dCBhbmQvb3IgbW9kaWZ5CiAgIGl0IHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUgR05VIEdlbmVyYWwg | UHVibGljIExpY2Vuc2UgYXMgcHVibGlzaGVkIGJ5CiAgIHRoZSBGcmVlIFNvZnR3YXJlIEZvdW5k | YXRpb247IGVpdGhlciB2ZXJzaW9uIDIgb2YgdGhlIExpY2Vuc2UsIG9yCiAgIChhdCB5b3VyIG9w | dGlvbikgYW55IGxhdGVyIHZlcnNpb24uIAoKICAgVGhpcyBwcm9ncmFtIGlzIGRpc3RyaWJ1dGVk | IGluIHRoZSBob3BlIHRoYXQgaXQgd2lsbCBiZSB1c2VmdWwsCiAgIGJ1dCBXSVRIT1VUIEFOWSBX | QVJSQU5UWTsgd2l0aG91dCBldmVuIHRoZSBpbXBsaWVkIHdhcnJhbnR5IG9mCiAgIE1FUkNIQU5U | QUJJTElUWSBvciBGSVRORVNTIEZPUiBBIFBBUlRJQ1VMQVIgUFVSUE9TRS4gIFNlZSB0aGUKICAg | R05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgZm9yIG1vcmUgZGV0YWlscy4KCiAgIFlvdSBzaG91 | bGQgaGF2ZSByZWNlaXZlZCBhIGNvcHkgb2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNl | CiAgIGFsb25nIHdpdGggdGhpcyBwcm9ncmFtOyBpZiBub3QsIHdyaXRlIHRvIHRoZSBGcmVlIFNv | ZnR3YXJlCiAgIEZvdW5kYXRpb24sIEluYy4sIDY3NSBNYXNzIEF2ZSwgQ2FtYnJpZGdlLCBNQSAw | MjEzOSwgVVNBLgoqLwoKI2luY2x1ZGUgPFB5dGhvbi5oPgojaW5jbHVkZSA8bWFnaWMuaD4KI2lu | Y2x1ZGUgInB5X21hZ2ljLmgiCgovKiBFeGNlcHRpb25zIHJhaXNlZCBieSB0aGlzIG1vZHVsZSAq | LwoKUHlPYmplY3QqIG1hZ2ljX2Vycm9yX29iajsKCi8qIENyZWF0ZSBhIG5ldyBtYWdpY19jb29r | aWVfaG5kIG9iamVjdCAqLwpQeU9iamVjdCogbmV3X21hZ2ljX2Nvb2tpZV9oYW5kbGUobWFnaWNf | dCBjb29raWUpCnsKICAgIG1hZ2ljX2Nvb2tpZV9obmQqIG1jaDsKCiAgICBtY2ggPSBQeU9iamVj | dF9OZXcobWFnaWNfY29va2llX2huZCwgJm1hZ2ljX2Nvb2tpZV90eXBlKTsKCiAgICBtY2gtPmNv | b2tpZSA9IGNvb2tpZTsKICAgIAogICAgcmV0dXJuIChQeU9iamVjdCopbWNoOwp9CgpzdGF0aWMg | Y2hhciBfbWFnaWNfb3Blbl9fZG9jX19bXSA9IAoiUmV0dXJucyBhIG1hZ2ljIGNvb2tpZSBvbiBz | dWNjZXNzIGFuZCBOb25lIG9uIGZhaWx1cmUuXG4iOwpzdGF0aWMgUHlPYmplY3QqIHB5X21hZ2lj | X29wZW4oUHlPYmplY3QqIHNlbGYsIFB5T2JqZWN0KiBhcmdzKQp7CiAgICBpbnQgZmxhZ3MgPSAw | OwogICAgbWFnaWNfdCBjb29raWU7CgogICAgaWYoIVB5QXJnX1BhcnNlVHVwbGUoYXJncywgImki | LCAmZmxhZ3MpKQogICAgICAgIHJldHVybiBOVUxMOwoKICAgIGlmKCEoY29va2llID0gbWFnaWNf | b3BlbihmbGFncykpKQogICAgICAgIHJldHVybiBOVUxMOwoKICAgIHJldHVybiBuZXdfbWFnaWNf | Y29va2llX2hhbmRsZShjb29raWUpOwp9CgpzdGF0aWMgY2hhciBfbWFnaWNfY2xvc2VfX2RvY19f | W10gPQoiQ2xvc2VzIHRoZSBtYWdpYyBkYXRhYmFzZSBhbmQgZGVhbGxvY2F0ZXMgYW55IHJlc291 | cmNlcyB1c2VkLlxuIjsKc3RhdGljIFB5T2JqZWN0KiBweV9tYWdpY19jbG9zZShQeU9iamVjdCog | c2VsZiwgUHlPYmplY3QqIGFyZ3MpCnsKICAgIG1hZ2ljX2Nvb2tpZV9obmQqIGhuZCA9IChtYWdp | Y19jb29raWVfaG5kKilzZWxmOwoKICAgIG1hZ2ljX2Nsb3NlKGhuZC0+Y29va2llKTsKICAgIAog | ICAgUHlfSU5DUkVGKFB5X05vbmUpOwogICAgcmV0dXJuIFB5X05vbmU7Cn0KCnN0YXRpYyBjaGFy | IF9tYWdpY19lcnJvcl9fZG9jX19bXSA9CiJSZXR1cm5zIGEgdGV4dHVhbCBleHBsYW5hdGlvbiBv | ZiB0aGUgbGFzdCBlcnJvciBvciBOb25lIFwKIGlmIHRoZXJlIHdhcyBubyBlcnJvci5cbiI7CnN0 | YXRpYyBQeU9iamVjdCogcHlfbWFnaWNfZXJyb3IoUHlPYmplY3QqIHNlbGYsIFB5T2JqZWN0KiBh | cmdzKQp7CiAgICBtYWdpY19jb29raWVfaG5kKiBobmQgPSAobWFnaWNfY29va2llX2huZCopc2Vs | ZjsKICAgIGNvbnN0IGNoYXIqIHJlc3VsdCA9IE5VTEw7CiAgICBQeU9iamVjdCogb2JqID0gTlVM | TDsKCiAgICByZXN1bHQgPSBtYWdpY19lcnJvcihobmQtPmNvb2tpZSk7CgogICAgaWYocmVzdWx0 | ICE9IE5VTEwpCiAgICAgICAgb2JqID0gUHlTdHJpbmdfRnJvbVN0cmluZyhyZXN1bHQpOwogICAg | ZWxzZSB7CiAgICAgICAgUHlfSU5DUkVGKFB5X05vbmUpOwogICAgICAgIG9iaiA9IFB5X05vbmU7 | CiAgICB9CgogICAgcmV0dXJuICBvYmo7Cn0KCnN0YXRpYyBjaGFyIF9tYWdpY19maWxlX19kb2Nf | X1tdID0gCiJSZXR1cm5zIGEgdGV4dHVhbCBkZXNjcmlwdGlvbiBvZiB0aGUgY29udGVudHMgb2Yg | dGhlIGFyZ3VtZW50IHBhc3NlZCBcCiBhcyBhIGZpbGVuYW1lIG9yIE5vbmUgaWYgYW4gZXJyb3Ig | b2NjdXJyZWQuXG4iOwpzdGF0aWMgUHlPYmplY3QqIHB5X21hZ2ljX2ZpbGUoUHlPYmplY3QqIHNl | bGYsIFB5T2JqZWN0KiBhcmdzKQp7CiAgICBtYWdpY19jb29raWVfaG5kKiBobmQgPSAobWFnaWNf | Y29va2llX2huZCopc2VsZjsKICAgIGNoYXIqIGZpbGVuYW1lID0gTlVMTDsKICAgIGNvbnN0IGNo | YXIqIHJlc3VsdCA9IE5VTEw7CiAgICBQeU9iamVjdCogb2JqID0gTlVMTDsKICAgIGludCBlcnIg | PSAwOwoKICAgIGlmKCEoUHlBcmdfUGFyc2VUdXBsZShhcmdzLCAicyIsICZmaWxlbmFtZSkpKQog | ICAgICAgIHJldHVybiBOVUxMOwoKICAgIHJlc3VsdCA9IG1hZ2ljX2ZpbGUoaG5kLT5jb29raWUs | IGZpbGVuYW1lKTsKCiAgICBpZihyZXN1bHQgIT0gTlVMTCkKICAgICAgICBvYmogPSBQeVN0cmlu | Z19Gcm9tU3RyaW5nKHJlc3VsdCk7CiAgICBlbHNlCiAgICAgICAgUHlFcnJfU2V0U3RyaW5nKG1h | Z2ljX2Vycm9yX29iaiwgImZhaWx1cmUgdHJ5aW5nIHRvIGRldGVybWluZSBmaWxlIHR5cGUiKTsK | CiAgICByZXR1cm4gb2JqOwp9CgpzdGF0aWMgY2hhciBfbWFnaWNfYnVmZmVyX19kb2NfX1tdID0K | IlJldHVybnMgYSB0ZXh0dWFsIGRlc2NyaXB0aW9uIG9mIHRoZSBjb250ZW50cyBvZiB0aGUgYXJn | dW1lbnQgcGFzc2VkIFwKIGFzIGEgYnVmZmVyIG9yIE5vbmUgaWYgYW4gZXJyb3Igb2NjdXJyZWQu | XG4iOwpzdGF0aWMgUHlPYmplY3QqIHB5X21hZ2ljX2J1ZmZlcihQeU9iamVjdCogc2VsZiwgUHlP | YmplY3QqIGFyZ3MpCnsKICAgIG1hZ2ljX2Nvb2tpZV9obmQqIGhuZCA9IChtYWdpY19jb29raWVf | aG5kKilzZWxmOwogICAgdm9pZCogYnVmZmVyID0gTlVMTDsKICAgIGludCBidWZmZXJfbGVuZ3Ro | ID0gMDsKICAgIGNvbnN0IGNoYXIqIHJlc3VsdCA9IE5VTEw7CiAgICBQeU9iamVjdCogb2JqID0g | TlVMTDsKCiAgICBpZighKFB5QXJnX1BhcnNlVHVwbGUoYXJncywgInMjIiwgKGNoYXIqKikmYnVm | ZmVyLCAmYnVmZmVyX2xlbmd0aCkpKQogICAgICAgIHJldHVybiBOVUxMOwoKICAgIHJlc3VsdCA9 | IG1hZ2ljX2J1ZmZlcihobmQtPmNvb2tpZSwgYnVmZmVyLCBidWZmZXJfbGVuZ3RoKTsKCiAgICBp | ZihyZXN1bHQgIT0gTlVMTCkKICAgICAgICBvYmogPSBQeVN0cmluZ19Gcm9tU3RyaW5nKHJlc3Vs | dCk7CiAgICBlbHNlCiAgICAgICAgUHlFcnJfU2V0U3RyaW5nKG1hZ2ljX2Vycm9yX29iaiwgImZh | aWx1cmUgdHJ5aW5nIHRvIGRldGVybWluZSBmaWxlIHR5cGUiKTsKCiAgICByZXR1cm4gb2JqOwp9 | CgpzdGF0aWMgY2hhciBfbWFnaWNfc2V0ZmxhZ3NfX2RvY19fW10gPQoiU2V0IGZsYWdzIG9uIHRo | ZSBjb29raWUgb2JqZWN0LlxuIFwKIFJldHVybnMgLTEgb24gc3lzdGVtcyB0aGF0IGRvbid0IHN1 | cHBvcnQgdXRpbWUoMikgb3IgdXRpbWVzKDIpIFwKIHdoZW4gTUFHSUNfUFJFU0VSVkVfQVRJTUUg | aXMgc2V0LlxuIjsKc3RhdGljIFB5T2JqZWN0KiBweV9tYWdpY19zZXRmbGFncyhQeU9iamVjdCog | c2VsZiwgUHlPYmplY3QqIGFyZ3MpCnsKICAgIG1hZ2ljX2Nvb2tpZV9obmQqIGhuZCA9IChtYWdp | Y19jb29raWVfaG5kKilzZWxmOwogICAgaW50IGZsYWdzOwogICAgaW50IHJlc3VsdDsKCiAgICBp | ZighKFB5QXJnX1BhcnNlVHVwbGUoYXJncywgImkiLCAmZmxhZ3MpKSkKICAgICAgICByZXR1cm4g | TlVMTDsKCiAgICByZXN1bHQgPSBtYWdpY19zZXRmbGFncyhobmQtPmNvb2tpZSwgZmxhZ3MpOwoK | ICAgIHJldHVybiBQeUludF9Gcm9tTG9uZyhyZXN1bHQpOwp9CgpzdGF0aWMgY2hhciBfbWFnaWNf | Y2hlY2tfX2RvY19fW10gPQoiQ2hlY2sgdGhlIHZhbGlkaXR5IG9mIGVudHJpZXMgaW4gdGhlIGNv | bG9uIHNlcGFyYXRlZCBsaXN0IG9mIGRhdGFiYXNlIGZpbGVzIFwKIHBhc3NlZCBhcyBhcmd1bWVu | dCBvciB0aGUgZGVmYXVsdCBkYXRhYmFzZSBmaWxlIGlmIG5vIGFyZ3VtZW50LlxuIFwKIFJldHVy | bnMgMCBvbiBzdWNjZXNzIGFuZCAtMSBvbiBmYWlsdXJlLlxuIjsKc3RhdGljIFB5T2JqZWN0KiBw | eV9tYWdpY19jaGVjayhQeU9iamVjdCogc2VsZiwgUHlPYmplY3QqIGFyZ3MpCnsKICAgIG1hZ2lj | X2Nvb2tpZV9obmQqIGhuZCA9IChtYWdpY19jb29raWVfaG5kKilzZWxmOwogICAgY2hhciogZmls | ZW5hbWUgPSBOVUxMOwogICAgaW50IHJlc3VsdDsKCiAgICBpZighKFB5QXJnX1BhcnNlVHVwbGUo | YXJncywgInxzIiwgJmZpbGVuYW1lKSkpCiAgICAgICAgcmV0dXJuIE5VTEw7CgogICAgcmVzdWx0 | ID0gbWFnaWNfY2hlY2soaG5kLT5jb29raWUsIGZpbGVuYW1lKTsKCiAgICByZXR1cm4gUHlJbnRf | RnJvbUxvbmcocmVzdWx0KTsKfQoKc3RhdGljIGNoYXIgX21hZ2ljX2NvbXBpbGVfX2RvY19fW10g | PQoiQ29tcGlsZSBlbnRyaWVzIGluIHRoZSBjb2xvbiBzZXBhcmF0ZWQgbGlzdCBvZiBkYXRhYmFz | ZSBmaWxlcyBcCiBwYXNzZWQgYXMgYXJndW1lbnQgb3IgdGhlIGRlZmF1bHQgZGF0YWJhc2UgZmls | ZSBpZiBubyBhcmd1bWVudC5cbiBcCiBSZXR1cm5zIDAgb24gc3VjY2VzcyBhbmQgLTEgb24gZmFp | bHVyZS5cbiBcCiBUaGUgY29tcGlsZWQgZmlsZXMgY3JlYXRlZCBhcmUgbmFtZWQgZnJvbSB0aGUg | YmFzZW5hbWUoMSkgb2YgZWFjaCBmaWxlIFwKIGFyZ3VtZW50IHdpdGggXCIubWdjXCIgYXBwZW5k | ZWQgdG8gaXQuXG4iOwpzdGF0aWMgUHlPYmplY3QqIHB5X21hZ2ljX2NvbXBpbGUoUHlPYmplY3Qq | IHNlbGYsIFB5T2JqZWN0KiBhcmdzKQp7CiAgICBtYWdpY19jb29raWVfaG5kKiBobmQgPSAobWFn | aWNfY29va2llX2huZCopc2VsZjsKICAgIGNoYXIqIGZpbGVuYW1lID0gTlVMTDsKICAgIGludCBy | ZXN1bHQ7CgogICAgaWYoIShQeUFyZ19QYXJzZVR1cGxlKGFyZ3MsICJ8cyIsICZmaWxlbmFtZSkp | KQogICAgICAgIHJldHVybiBOVUxMOwoKICAgIHJlc3VsdCA9IG1hZ2ljX2NvbXBpbGUoaG5kLT5j | b29raWUsIGZpbGVuYW1lKTsKCiAgICByZXR1cm4gUHlJbnRfRnJvbUxvbmcocmVzdWx0KTsKfQoK | c3RhdGljIGNoYXIgX21hZ2ljX2xvYWRfX2RvY19fW10gPQoiTXVzdCBiZSB1c2VkIHRvIGxvYWQg | ZW50cmllcyBpbiB0aGUgY29sb24gc2VwYXJhdGVkIGxpc3Qgb2YgZGF0YWJhc2UgZmlsZXMgXAog | cGFzc2VkIGFzIGFyZ3VtZW50IG9yIHRoZSBkZWZhdWx0IGRhdGFiYXNlIGZpbGUgaWYgbm8gYXJn | dW1lbnQgYmVmb3JlIFwKIGFueSBtYWdpYyBxdWVyaWVzIGNhbiBiZSBwZXJmb3JtZWQuXG4gXAog | UmV0dXJucyAwIG9uIHN1Y2Nlc3MgYW5kIC0xIG9uIGZhaWx1cmUuXG4iOwpzdGF0aWMgUHlPYmpl | Y3QqIHB5X21hZ2ljX2xvYWQoUHlPYmplY3QqIHNlbGYsIFB5T2JqZWN0KiBhcmdzKQp7CiAgICBt | YWdpY19jb29raWVfaG5kKiBobmQgPSAobWFnaWNfY29va2llX2huZCopc2VsZjsKICAgIGNoYXIq | IGZpbGVuYW1lID0gTlVMTDsKICAgIGludCByZXN1bHQ7CgogICAgaWYoIShQeUFyZ19QYXJzZVR1 | cGxlKGFyZ3MsICJ8cyIsICZmaWxlbmFtZSkpKQogICAgICAgIHJldHVybiBOVUxMOwoKICAgIHJl | c3VsdCA9IG1hZ2ljX2xvYWQoaG5kLT5jb29raWUsIGZpbGVuYW1lKTsKCiAgICByZXR1cm4gUHlJ | bnRfRnJvbUxvbmcocmVzdWx0KTsKfQoKLyogb2JqZWN0IG1ldGhvZHMgKi8KCnN0YXRpYyBQeU1l | dGhvZERlZiBtYWdpY19jb29raWVfaG5kX21ldGhvZHNbXSA9IHsKICAgIHsgImNsb3NlIiwgKFB5 | Q0Z1bmN0aW9uKXB5X21hZ2ljX2Nsb3NlLAogICAgICBNRVRIX05PQVJHUywgX21hZ2ljX2Nsb3Nl | X19kb2NfXyB9LAogICAgeyAiZXJyb3IiLCAoUHlDRnVuY3Rpb24pcHlfbWFnaWNfZXJyb3IsCiAg | ICAgIE1FVEhfTk9BUkdTLCBfbWFnaWNfZXJyb3JfX2RvY19fIH0sCiAgICB7ICJmaWxlIiwgKFB5 | Q0Z1bmN0aW9uKXB5X21hZ2ljX2ZpbGUsCiAgICAgIE1FVEhfVkFSQVJHUywgX21hZ2ljX2ZpbGVf | X2RvY19fIH0sCiAgICB7ICJidWZmZXIiLCAoUHlDRnVuY3Rpb24pcHlfbWFnaWNfYnVmZmVyLAog | ICAgICBNRVRIX1ZBUkFSR1MsIF9tYWdpY19idWZmZXJfX2RvY19fIH0sCiAgICB7ICJzZXRmbGFn | cyIsIChQeUNGdW5jdGlvbilweV9tYWdpY19zZXRmbGFncywKICAgICAgTUVUSF9WQVJBUkdTLCBf | bWFnaWNfc2V0ZmxhZ3NfX2RvY19fIH0sCiAgICB7ICJjaGVjayIsIChQeUNGdW5jdGlvbilweV9t | YWdpY19jaGVjaywKICAgICAgTUVUSF9WQVJBUkdTLCBfbWFnaWNfY2hlY2tfX2RvY19fIH0sCiAg | ICB7ICJjb21waWxlIiwgKFB5Q0Z1bmN0aW9uKXB5X21hZ2ljX2NvbXBpbGUsCiAgICAgIE1FVEhf | VkFSQVJHUywgX21hZ2ljX2NvbXBpbGVfX2RvY19fIH0sCiAgICB7ICJsb2FkIiwgKFB5Q0Z1bmN0 | aW9uKXB5X21hZ2ljX2xvYWQsCiAgICAgIE1FVEhfVkFSQVJHUywgX21hZ2ljX2xvYWRfX2RvY19f | IH0sCiAgICB7IE5VTEwsIE5VTEwgfQp9OwoKLyogbW9kdWxlIGxldmVsIG1ldGhvZHMgKi8KCnN0 | YXRpYyBQeU1ldGhvZERlZiBtYWdpY19tZXRob2RzW10gPSB7CiAgICB7ICJvcGVuIiwgKFB5Q0Z1 | bmN0aW9uKXB5X21hZ2ljX29wZW4sCiAgICAgIE1FVEhfVkFSQVJHUywgX21hZ2ljX29wZW5fX2Rv | Y19fIH0sCiAgICB7IE5VTEwsIE5VTEwgfQp9OwoKc3RhdGljIHZvaWQgcHlfbWFnaWNfZGVhbGxv | YyhQeU9iamVjdCogc2VsZikKewogICAgUHlPYmplY3RfRGVsKHNlbGYpOwp9CgpzdGF0aWMgUHlP | YmplY3QqIHB5X21hZ2ljX2dldGF0dHIoUHlPYmplY3QqIHNlbGYsIGNoYXIqIGF0dHJuYW1lKQp7 | CiAgICByZXR1cm4gUHlfRmluZE1ldGhvZChtYWdpY19jb29raWVfaG5kX21ldGhvZHMsIHNlbGYs | IGF0dHJuYW1lKTsKfQoKUHlUeXBlT2JqZWN0IG1hZ2ljX2Nvb2tpZV90eXBlID0gewogICAgUHlP | YmplY3RfSEVBRF9JTklUKE5VTEwpCiAgICAwLAogICAgIk1hZ2ljIGNvb2tpZSIsCiAgICBzaXpl | b2YobWFnaWNfY29va2llX2huZCksCiAgICAwLAogICAgcHlfbWFnaWNfZGVhbGxvYywgLyogdHBf | ZGVhbGxvYyAqLwogICAgMCwgICAgICAgICAgICAgICAgLyogdHBfcHJpbnQgKi8KICAgIHB5X21h | Z2ljX2dldGF0dHIsIC8qIHRwX2dldGF0dHIgKi8KICAgIDAsICAgICAgICAgICAgICAgIC8qIHRw | X3NldGF0dHIgKi8KICAgIDAsICAgICAgICAgICAgICAgIC8qIHRwX2NvbXBhcmUgKi8KICAgIDAs | ICAgICAgICAgICAgICAgIC8qIHRwX3JlcHIgKi8KICAgIDAsICAgICAgICAgICAgICAgIC8qIHRw | X2FzX251bWJlciAqLwogICAgMCwgICAgICAgICAgICAgICAgLyogdHBfYXNfc2VxdWVuY2UgKi8K | ICAgIDAsICAgICAgICAgICAgICAgIC8qIHRwX2FzX21hcHBpbmcgKi8KICAgIDAsICAgICAgICAg | ICAgICAgIC8qIHRwX2hhc2ggKi8KfTsKCi8qIEluaXRpYWxpemUgY29uc3RhbnRzICovCgpzdGF0 | aWMgc3RydWN0IGNvbnN0X3ZhbHMgewogICAgY29uc3QgY2hhciogY29uc3QgbmFtZTsKICAgIHVu | c2lnbmVkIGludCB2YWx1ZTsKfSBtb2R1bGVfY29uc3RfdmFsc1tdID0gewogICAgeyAiTUFHSUNf | Tk9ORSIsIE1BR0lDX05PTkUgfSwKICAgIHsgIk1BR0lDX0RFQlVHIiwgTUFHSUNfREVCVUcgfSwK | ICAgIHsgIk1BR0lDX1NZTUxJTksiLCBNQUdJQ19TWU1MSU5LIH0sCiAgICB7ICJNQUdJQ19DT01Q | UkVTUyIsIE1BR0lDX0NPTVBSRVNTIH0sCiAgICB7ICJNQUdJQ19ERVZJQ0VTIiwgTUFHSUNfREVW | SUNFUyB9LAogICAgeyAiTUFHSUNfTUlNRSIsIE1BR0lDX01JTUUgfSwKICAgIHsgIk1BR0lDX0NP | TlRJTlVFIiwgTUFHSUNfQ09OVElOVUUgfSwKICAgIHsgIk1BR0lDX0NIRUNLIiwgTUFHSUNfQ0hF | Q0sgfSwKICAgIHsgIk1BR0lDX1BSRVNFUlZFX0FUSU1FIiwgTUFHSUNfUFJFU0VSVkVfQVRJTUUg | fSwKICAgIHsgTlVMTCB9Cn07CgpzdGF0aWMgdm9pZCBjb25zdF9pbml0KFB5T2JqZWN0KiBkaWN0 | KQp7CiAgICBzdHJ1Y3QgY29uc3RfdmFscyogdG1wOwogICAgUHlPYmplY3QgKm9iajsKCiAgICBm | b3IodG1wID0gbW9kdWxlX2NvbnN0X3ZhbHM7IHRtcC0+bmFtZTsgKyt0bXApIHsKICAgICAgICBv | YmogPSBQeUludF9Gcm9tTG9uZyh0bXAtPnZhbHVlKTsKICAgICAgICBQeURpY3RfU2V0SXRlbVN0 | cmluZyhkaWN0LCB0bXAtPm5hbWUsIG9iaik7CiAgICAgICAgUHlfREVDUkVGKG9iaik7CiAgICB9 | Cn0KCi8qCiAqIE1vZHVsZSBpbml0aWFsaXphdGlvbgogKi8KCnZvaWQgaW5pdG1hZ2ljKHZvaWQp | CnsKICAgIFB5T2JqZWN0KiBtb2R1bGU7CiAgICBQeU9iamVjdCogZGljdDsKCiAgICAvKiBJbml0 | aWFsaXplIG1vZHVsZSAqLwogICAgCiAgICBtb2R1bGUgPSBQeV9Jbml0TW9kdWxlKCJtYWdpYyIs | IG1hZ2ljX21ldGhvZHMpOwogICAgZGljdCA9IFB5TW9kdWxlX0dldERpY3QobW9kdWxlKTsKCiAg | ICBtYWdpY19lcnJvcl9vYmogPSBQeUVycl9OZXdFeGNlcHRpb24oIm1hZ2ljLmVycm9yIiwgTlVM | TCwgTlVMTCk7CiAgICBQeURpY3RfU2V0SXRlbVN0cmluZyhkaWN0LCAiZXJyb3IiLCBtYWdpY19l | cnJvcl9vYmopOwoKICAgIG1hZ2ljX2Nvb2tpZV90eXBlLm9iX3R5cGUgPSAmUHlUeXBlX1R5cGU7 | CgogICAgLyogSW5pdGlhbGl6ZSBjb25zdGFudHMgKi8KCiAgICBjb25zdF9pbml0KGRpY3QpOwoK | ICAgIGlmKFB5RXJyX09jY3VycmVkKCkpCiAgICAgICAgUHlfRmF0YWxFcnJvcigiY2FuJ3QgaW5p | dGlhbGl6ZSBtb2R1bGUgbWFnaWMiKTsKfQo= | | ------=_20031007145726_30296-- | | _______________________________________________ | File mailing list | File@mx.gw.com | http://mx.gw.com/mailman/listinfo/file -- End of excerpt from "Brett Funderburg" From brettf at deepfile.com Wed Oct 8 11:09:33 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:56 2005 Subject: Segfault in magic_check Message-ID: <1064.192.168.123.158.1065625773.squirrel@staff.deepfile.com> Maybe some of these apis aren't quite finised yet but the following: m = magic_open(MAGIC_NONE) magic_load(m, NULL) magic_check(m, NULL) causes a segfault. According to libmagic(3) passing NULL as the filename pointer to magic_check should cause the default database to be checked but it doesn't look like the code in apprentice.c/file_apprentice is set up to handle this. Backtrace looks like this: #0 0x4207a703 in strlen () from /lib/tls/libc.so.6 #1 0x4207a445 in strdup () from /lib/tls/libc.so.6 #2 0x40417636 in file_apprentice (ms=0x8185218, fn=0x0, action=1) at apprentice.c:204 #3 0x404170be in magic_check (ms=0x0, magicfile=0x0) at magic.c:188 #4 0x4001d2b7 in py_magic_check (self=0x0, args=0x4011d02c) at py_magic.c:172 ...python call stack omitted... As a point of reference, this works fine: m = magic_open(MAGIC_NONE) magic_load(m, NULL) magic_file(m, "/path/to/some/file") and so does this: m = magic_open(MAGIC_NONE) magic_load(m, NULL) magic_buffer(m, some_data_buffer, buffer_size) brett From brettf at deepfile.com Wed Oct 8 11:35:12 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:56 2005 Subject: Python bindings [py_magic.h] 1 of 4 Message-ID: <1781.192.168.123.158.1065627312.squirrel@staff.deepfile.com> -- Begin -- /* Python wrappers for magic functions. Copyright (C) Brett Funderburg, Deepfile Corp. Austin, TX, US 2003 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ #ifndef _PY_MAGIC_H #define _PY_MAGIC_H typedef struct { PyObject_HEAD magic_t cookie; } magic_cookie_hnd; #endif /* _PY_MAGIC_H */ -- End -- From brettf at deepfile.com Wed Oct 8 11:40:34 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:56 2005 Subject: Python bindings [py_magic.c] 2 of 4 Message-ID: <1917.192.168.123.158.1065627634.squirrel@staff.deepfile.com> /* Python wrappers for magic functions. Copyright (C) Brett Funderburg, Deepfile Corp. Austin, TX, US 2003 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ #include #include #include "py_magic.h" /* Exceptions raised by this module */ PyObject* magic_error_obj; /* Create a new magic_cookie_hnd object */ PyObject* new_magic_cookie_handle(magic_t cookie) { magic_cookie_hnd* mch; mch = PyObject_New(magic_cookie_hnd, &magic_cookie_type); mch->cookie = cookie; return (PyObject*)mch; } static char _magic_open__doc__[] = "Returns a magic cookie on success and None on failure.\n"; static PyObject* py_magic_open(PyObject* self, PyObject* args) { int flags = 0; magic_t cookie; if(!PyArg_ParseTuple(args, "i", &flags)) return NULL; if(!(cookie = magic_open(flags))) { PyErr_SetString(magic_error_obj, "failure initializing magic cookie"); return NULL; } return new_magic_cookie_handle(cookie); } static char _magic_close__doc__[] = "Closes the magic database and deallocates any resources used.\n"; static PyObject* py_magic_close(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; magic_close(hnd->cookie); Py_INCREF(Py_None); return Py_None; } static char _magic_error__doc__[] = "Returns a textual explanation of the last error or None \ if there was no error.\n"; static PyObject* py_magic_error(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; const char* message = NULL; PyObject* result = NULL; message = magic_error(hnd->cookie); if(message != NULL) result = PyString_FromString(message); else { Py_INCREF(Py_None); result = Py_None; } return result; } static char _magic_file__doc__[] = "Returns a textual description of the contents of the argument passed \ as a filename or None if an error occurred.\n"; static PyObject* py_magic_file(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; char* filename = NULL; const char* message = NULL; PyObject* result = NULL; if(!(PyArg_ParseTuple(args, "s", &filename))) return NULL; message = magic_file(hnd->cookie, filename); if(message != NULL) result = PyString_FromString(message); else PyErr_SetString(PyExc_RuntimeError, "failure determining file type"); return result; } static char _magic_buffer__doc__[] = "Returns a textual description of the contents of the argument passed \ as a buffer or None if an error occurred.\n"; static PyObject* py_magic_buffer(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; void* buffer = NULL; int buffer_length = 0; const char* message = NULL; PyObject* result = NULL; if(!(PyArg_ParseTuple(args, "s#", (char**)&buffer, &buffer_length))) return NULL; message = magic_buffer(hnd->cookie, buffer, buffer_length); if(message != NULL) result = PyString_FromString(message); else PyErr_SetString(PyExc_RuntimeError, "failure determining buffer type"); return result; } static char _magic_setflags__doc__[] = "Set flags on the cookie object.\n \ Returns -1 on systems that don't support utime(2) or utimes(2) \ when MAGIC_PRESERVE_ATIME is set.\n"; static PyObject* py_magic_setflags(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; int flags; int result; if(!(PyArg_ParseTuple(args, "i", &flags))) return NULL; result = magic_setflags(hnd->cookie, flags); return PyInt_FromLong(result); } static char _magic_check__doc__[] = "Check the validity of entries in the colon separated list of database files \ passed as argument or the default database file if no argument.\n \ Returns 0 on success and -1 on failure.\n"; static PyObject* py_magic_check(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; char* filename = NULL; int result; if(!(PyArg_ParseTuple(args, "|s", &filename))) return NULL; result = magic_check(hnd->cookie, filename); return PyInt_FromLong(result); } static char _magic_compile__doc__[] = "Compile entries in the colon separated list of database files \ passed as argument or the default database file if no argument.\n \ Returns 0 on success and -1 on failure.\n \ The compiled files created are named from the basename(1) of each file \ argument with \".mgc\" appended to it.\n"; static PyObject* py_magic_compile(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; char* filename = NULL; int result; if(!(PyArg_ParseTuple(args, "|s", &filename))) return NULL; result = magic_compile(hnd->cookie, filename); return PyInt_FromLong(result); } static char _magic_load__doc__[] = "Must be used to load entries in the colon separated list of database files \ passed as argument or the default database file if no argument before \ any magic queries can be performed.\n \ Returns 0 on success and -1 on failure.\n"; static PyObject* py_magic_load(PyObject* self, PyObject* args) { magic_cookie_hnd* hnd = (magic_cookie_hnd*)self; char* filename = NULL; int result; if(!(PyArg_ParseTuple(args, "|s", &filename))) return NULL; result = magic_load(hnd->cookie, filename); return PyInt_FromLong(result); } /* object methods */ static PyMethodDef magic_cookie_hnd_methods[] = { { "close", (PyCFunction)py_magic_close, METH_NOARGS, _magic_close__doc__ }, { "error", (PyCFunction)py_magic_error, METH_NOARGS, _magic_error__doc__ }, { "file", (PyCFunction)py_magic_file, METH_VARARGS, _magic_file__doc__ }, { "buffer", (PyCFunction)py_magic_buffer, METH_VARARGS, _magic_buffer__doc__ }, { "setflags", (PyCFunction)py_magic_setflags, METH_VARARGS, _magic_setflags__doc__ }, { "check", (PyCFunction)py_magic_check, METH_VARARGS, _magic_check__doc__ }, { "compile", (PyCFunction)py_magic_compile, METH_VARARGS, _magic_compile__doc__ }, { "load", (PyCFunction)py_magic_load, METH_VARARGS, _magic_load__doc__ }, { NULL, NULL } }; /* module level methods */ static PyMethodDef magic_methods[] = { { "open", (PyCFunction)py_magic_open, METH_VARARGS, _magic_open__doc__ }, { NULL, NULL } }; static void py_magic_dealloc(PyObject* self) { PyObject_Del(self); } static PyObject* py_magic_getattr(PyObject* self, char* attrname) { return Py_FindMethod(magic_cookie_hnd_methods, self, attrname); } PyTypeObject magic_cookie_type = { PyObject_HEAD_INIT(NULL) 0, "Magic cookie", sizeof(magic_cookie_hnd), 0, py_magic_dealloc, /* tp_dealloc */ 0, /* tp_print */ py_magic_getattr, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ }; /* Initialize constants */ static struct const_vals { const char* const name; unsigned int value; } module_const_vals[] = { { "MAGIC_NONE", MAGIC_NONE }, { "MAGIC_DEBUG", MAGIC_DEBUG }, { "MAGIC_SYMLINK", MAGIC_SYMLINK }, { "MAGIC_COMPRESS", MAGIC_COMPRESS }, { "MAGIC_DEVICES", MAGIC_DEVICES }, { "MAGIC_MIME", MAGIC_MIME }, { "MAGIC_CONTINUE", MAGIC_CONTINUE }, { "MAGIC_CHECK", MAGIC_CHECK }, { "MAGIC_PRESERVE_ATIME", MAGIC_PRESERVE_ATIME }, { NULL } }; static void const_init(PyObject* dict) { struct const_vals* tmp; PyObject *obj; for(tmp = module_const_vals; tmp->name; ++tmp) { obj = PyInt_FromLong(tmp->value); PyDict_SetItemString(dict, tmp->name, obj); Py_DECREF(obj); } } /* * Module initialization */ void initmagic(void) { PyObject* module; PyObject* dict; /* Initialize module */ module = Py_InitModule("magic", magic_methods); dict = PyModule_GetDict(module); magic_error_obj = PyErr_NewException("magic.error", NULL, NULL); PyDict_SetItemString(dict, "error", magic_error_obj); magic_cookie_type.ob_type = &PyType_Type; /* Initialize constants */ const_init(dict); if(PyErr_Occurred()) Py_FatalError("can't initialize module magic"); } From brettf at deepfile.com Wed Oct 8 11:41:24 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: Python bindings [setup.py] 3 of 4 Message-ID: <1941.192.168.123.158.1065627684.squirrel@staff.deepfile.com> from distutils.core import setup, Extension magic_module = Extension('magic', libraries = ['magic'], library_dirs = ['./','../','/usr/lib/'], include_dirs = ['./','../','/usr/include/'], sources = ['py_magic.c']) setup (name = 'Magic file extensions', version = '0.1', author = 'Brett Funderburg', author_email = 'brettf@deepfile.com', license = 'GPL', description = 'libmagic python bindings', ext_modules = [magic_module]) From brettf at deepfile.com Wed Oct 8 11:42:21 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: Python bindings [README] 4 of 4 Message-ID: <1967.192.168.123.158.1065627741.squirrel@staff.deepfile.com> This directory contains Python bindings to allow you to access the libmagic api. At the moment their status is "experimental" and they are not built by default. In order to be able to compile magic-python you need to have python and the python-dev packages installed. Python libraries are always built for a particular version of Python (2.2, 2.3, etc), and libraries built for one version will not be seen by another. To build: $ python setup.py build Now, you can install the modules: $ cp build/lib.*/magic.so /usr/lib/python2.3/lib-dynload/ (the directory /usr/lib/python2.3 may vary, depending on your installation) magic-python should work now! From brettf at deepfile.com Wed Oct 8 11:49:25 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: Python bindings notes Message-ID: <2312.192.168.123.158.1065628165.squirrel@staff.deepfile.com> To build, make a directory called 'python' in 'src' and then follow the instructions in the README. Unless something got screwed up in the cut and paste, the code should compile cleanly and without warnings. You may need to tinker with the library and include paths specified in the distutils build script (setup.py) to make things work on your machine. The following python script should give you an idea of how things work. -- begin -- import magic c = magic.open(magic.MAGIC_NONE) c.load() c.file("/path/to/some/file") f = file("/path/to/some/file", "r") buffer = f.read(4096) f.close() c.buffer(buffer) c.close() --end-- Regards, brett From christos at zoulas.com Wed Oct 8 13:23:54 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: Segfault in magic_check In-Reply-To: <1064.192.168.123.158.1065625773.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 8, 10:09am) Message-ID: <20031008172354.726667E05@beowulf.gw.com> On Oct 8, 10:09am, brettf@deepfile.com ("Brett Funderburg") wrote: -- Subject: Segfault in magic_check No, they are just buggy :-) Try file-4.05... christos | Maybe some of these apis aren't quite finised yet but the following: | | m = magic_open(MAGIC_NONE) | magic_load(m, NULL) | magic_check(m, NULL) | | causes a segfault. From brettf at deepfile.com Wed Oct 8 14:00:45 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: Segfault in magic_check In-Reply-To: <20031008172354.726667E05@beowulf.gw.com> References: <1064.192.168.123.158.1065625773.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 8, 10:09am) <20031008172354.726667E05@beowulf.gw.com> Message-ID: <2144.192.168.123.158.1065636045.squirrel@staff.deepfile.com> Works - no segfault. > On Oct 8, 10:09am, brettf@deepfile.com ("Brett Funderburg") wrote: > -- Subject: Segfault in magic_check > > No, they are just buggy :-) > > Try file-4.05... > > christos > > | Maybe some of these apis aren't quite finised yet but the following: > | > | m = magic_open(MAGIC_NONE) > | magic_load(m, NULL) > | magic_check(m, NULL) > | > | causes a segfault. > _______________________________________________ > File mailing list > File@mx.gw.com > http://mx.gw.com/mailman/listinfo/file > From brettf at deepfile.com Tue Oct 14 12:02:49 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: comments on libmagic Message-ID: <2064.192.168.123.158.1066147369.squirrel@staff.deepfile.com> I've been playing around with libmagic over the past few days and I'd like to offer the following comments for feedback. I've noticed some inconsistencies in how the api works. For example, Case 1 (pseudo-code) magic_open() result = magic_file("/path/to/some/file") if !result print magic_error() <- "No magic files loaded" Case 2 magic_open() magic_load() result = magic_file("/file/not/found") print magic_error() <- "" print result <- "Can't stat file..." Case 1 works as I would expect it to. I think it's preferable in Case 2 to have it work like the following: magic_open() magic_load() result = magic_file("/file/not/found") if !result print magic_error() <- "Can't stat file" Indeed, I think this is how the man page describes how it should work. However, this points out an additional limitation. Relying on simple 0, -1, NULL return values from library functions does not provide enough information to the caller about what has gone wrong. Textual error descriptions provided by magic_error are fine but aren't suitable for error handling. As a consumer of the libmagic api, this is what I would like to do: my_app_get_file_type(char* file_path) { ... set up libmagic stuff ... result = magic_file(file_path); if(result != NULL) return result; <- "ELF executable, etc." error = magic_errno() switch(error) { case STAT_ERROR: ... handle this case ... print magic_error() <- "Doesn't exist..." case OPEN_ERROR: ... handle this other case ... print magic_error() <- "Can't open..." } } Comments? brett From christos at zoulas.com Tue Oct 14 15:36:07 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: comments on libmagic In-Reply-To: <2064.192.168.123.158.1066147369.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 14, 11:02am) Message-ID: <20031014193607.EC41D7E05@beowulf.gw.com> On Oct 14, 11:02am, brettf@deepfile.com ("Brett Funderburg") wrote: -- Subject: comments on libmagic | I've been playing around with libmagic over the past few days and I'd like | to offer the following comments for feedback. | | I've noticed some inconsistencies in how the api works. For example, | | Case 1 | | (pseudo-code) | | magic_open() | result = magic_file("/path/to/some/file") | if !result | print magic_error() <- "No magic files loaded" Yes, this is because this is an internal magic error. | Case 2 | | magic_open() | magic_load() | result = magic_file("/file/not/found") | print magic_error() <- "" | print result <- "Can't stat file..." Yes, because this is what file(1) is supposed to return. Actually by posix it is supposed to return "not found". | Case 1 works as I would expect it to. I think it's preferable in Case 2 to | have it work like the following: | | magic_open() | magic_load() | result = magic_file("/file/not/found") | if !result | print magic_error() <- "Can't stat file" | | Indeed, I think this is how the man page describes how it should work. | However, this points out an additional limitation. Relying on simple 0, | -1, NULL return values from library functions does not provide enough | information to the caller about what has gone wrong. Textual error | descriptions provided by magic_error are fine but aren't suitable for | error handling. As a consumer of the libmagic api, this is what I would | like to do: | | my_app_get_file_type(char* file_path) { | | ... set up libmagic stuff ... | | result = magic_file(file_path); | if(result != NULL) | return result; <- "ELF executable, etc." | | error = magic_errno() | switch(error) { | case STAT_ERROR: | ... handle this case ... | print magic_error() <- "Doesn't exist..." | | case OPEN_ERROR: | ... handle this other case ... | print magic_error() <- "Can't open..." | } | } | | Comments? | | brett I see your point. I think that the best solution is to give you access to the errno information, and change the default behavior via a flag: ms = magic_open(MAGIG_ERROR); if ((result = magic_file(ms, fn)) == NULL) { int error = magic_errno(ms); if (error == 0) // File internal error fprintf("%s: %s\n", fn, file_error(ms)); else // Os error fprintf("%s: %s\n", fn, strerror(error)); } In fact, I have implemented all this already... christos From brettf at deepfile.com Tue Oct 14 16:28:34 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: comments on libmagic In-Reply-To: <20031014193607.EC41D7E05@beowulf.gw.com> References: <2064.192.168.123.158.1066147369.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 14, 11:02am) <20031014193607.EC41D7E05@beowulf.gw.com> Message-ID: <1705.192.168.123.158.1066163314.squirrel@staff.deepfile.com> > On Oct 14, 11:02am, brettf@deepfile.com ("Brett Funderburg") wrote: > -- Subject: comments on libmagic > -snip- > | > | Case 1 > | > | (pseudo-code) > | > | magic_open() > | result = magic_file("/path/to/some/file") > | if !result > | print magic_error() <- "No magic files loaded" > > Yes, this is because this is an internal magic error. Right, I purposely used it as an example of "correct" behavior. > > | Case 2 > | > | magic_open() > | magic_load() > | result = magic_file("/file/not/found") > | print magic_error() <- "" > | print result <- "Can't stat file..." > > Yes, because this is what file(1) is supposed to return. Actually by posix > it is supposed to return "not found". Right, I understand. My point was that file(1) (or any other user of libmagic) should understand how it should communicate errors to the user. I see from your response below that you understood my point. - snip - > I see your point. I think that the best solution is to give you access to > the errno information, Exactly. > and change the default behavior via a flag: Yes, this seems like the best way to preserve backward api compatibility for now. > > ms = magic_open(MAGIG_ERROR); > > if ((result = magic_file(ms, fn)) == NULL) { > int error = magic_errno(ms); > if (error == 0) > // File internal error > fprintf("%s: %s\n", fn, file_error(ms)); > else > // Os error > fprintf("%s: %s\n", fn, strerror(error)); > } > > In fact, I have implemented all this already... > In my example, I used a hypothetical new api (magic_errno) to get the integer value of the error. You're suggesting changing the default behavior via a flag, yet your code example above also uses magic_errno. Did you intend for this to be 'magic_error' instead? Using your example, how would I know whether the call to magic_file succeeded or not? With the current libmagic, giving a path to a non-existent file still results in a non-NULL value being assigned to 'result'. This gives the appearance that the call succeeded. I could see two potential problems: * unable to determine whether magic_file succeeded or failed based on return value alone. * unable to determine whether magic_file succeeded by calling magic_error since there is no 'success' error code. (0 = internal error, non-0 = OS error) What am I missing? brett From christos at zoulas.com Tue Oct 14 17:51:58 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: comments on libmagic In-Reply-To: <1705.192.168.123.158.1066163314.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 14, 3:28pm) Message-ID: <20031014215158.E9A3D7E05@beowulf.gw.com> On Oct 14, 3:28pm, brettf@deepfile.com ("Brett Funderburg") wrote: -- Subject: Re: comments on libmagic | Yes, this seems like the best way to preserve backward api compatibility | for now. | | > | > ms = magic_open(MAGIG_ERROR); | > | > if ((result = magic_file(ms, fn)) == NULL) { | > int error = magic_errno(ms); | > if (error == 0) | > // File internal error | > fprintf("%s: %s\n", fn, magic_error(ms)); | > else | > // Os error | > fprintf("%s: %s\n", fn, strerror(error)); | > } | > | > In fact, I have implemented all this already... | > | | In my example, I used a hypothetical new api (magic_errno) to get the | integer value of the error. You're suggesting changing the default | behavior via a flag, yet your code example above also uses magic_errno. | Did you intend for this to be 'magic_error' instead? No, that was what I intended. magic_errno() always returns the system's errno value if the error was caused by a system related error and 0 if it was an error internal to libmagic. | Using your example, how would I know whether the call to magic_file | succeeded or not? With the current libmagic, giving a path to a | non-existent file still results in a non-NULL value being assigned to | 'result'. This gives the appearance that the call succeeded. I could see | two potential problems: | | * unable to determine whether magic_file succeeded or failed based on | return value alone. | * unable to determine whether magic_file succeeded by calling magic_error | since there is no 'success' error code. (0 = internal error, non-0 = OS | error) | | What am I missing? That I am passing MAGIC_ERROR ro magic_open, and this changes the default behavior of magic_file() to return NULL on system related errors. This is a new flag that I just added, that causes magic_file() to return NULL in the cases you want to handle the error yourself. christos From brettf at deepfile.com Tue Oct 14 18:15:49 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: comments on libmagic In-Reply-To: <20031014215158.E9A3D7E05@beowulf.gw.com> References: <1705.192.168.123.158.1066163314.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 14, 3:28pm) <20031014215158.E9A3D7E05@beowulf.gw.com> Message-ID: <4858.192.168.123.158.1066169749.squirrel@staff.deepfile.com> > On Oct 14, 3:28pm, brettf@deepfile.com ("Brett Funderburg") wrote: > | > | In my example, I used a hypothetical new api (magic_errno) to get the > | integer value of the error. You're suggesting changing the default > | behavior via a flag, yet your code example above also uses magic_errno. > | Did you intend for this to be 'magic_error' instead? > > No, that was what I intended. magic_errno() always returns the system's > errno value if the error was caused by a system related error and 0 > if it was an error internal to libmagic. So magic_errno is a new api for libmagic? > | > | What am I missing? > > That I am passing MAGIC_ERROR ro magic_open, and this changes the default > behavior of magic_file() to return NULL on system related errors. This is > a new flag that I just added, that causes magic_file() to return NULL in > the cases you want to handle the error yourself. Ok, that makes it clear now. Thanks. When will these changes be available for download? brett From christos at zoulas.com Tue Oct 14 20:09:24 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: comments on libmagic In-Reply-To: <4858.192.168.123.158.1066169749.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 14, 5:15pm) Message-ID: <20031015000924.A2E797E03@beowulf.gw.com> On Oct 14, 5:15pm, brettf@deepfile.com ("Brett Funderburg") wrote: -- Subject: Re: comments on libmagic | > No, that was what I intended. magic_errno() always returns the system's | > errno value if the error was caused by a system related error and 0 | > if it was an error internal to libmagic. | | So magic_errno is a new api for libmagic? Yes. | > That I am passing MAGIC_ERROR ro magic_open, and this changes the default | > behavior of magic_file() to return NULL on system related errors. This is | > a new flag that I just added, that causes magic_file() to return NULL in | > the cases you want to handle the error yourself. | | Ok, that makes it clear now. Thanks. When will these changes be available | for download? | | brett Soon... I need to clean up a few more things, but really soon now. christos From christos at zoulas.com Tue Oct 14 22:42:04 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: file 4.06 is now available Message-ID: <20031015024204.7DAED7E09@beowulf.gw.com> >From ftp.astron.com:/pub/file Enjoy, christos 2003-10-14 21:39 Christos Zoulas * Python bindings (Brett Funderburg) * Don't lookup past the end of the buffer (Chad Hanson) * Add MAGIC_ERROR and api on magic_errno() From brettf at deepfile.com Thu Oct 16 15:09:40 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: Memory leak in apprentice.c Message-ID: <3104.192.168.123.158.1066331380.squirrel@staff.deepfile.com> Using valgrind --leak-check=yes on file-4.06, it reports that apprentice.c line 227 is leaking memory. It appears in this bit of code: if ((fn = mfn = strdup(fn)) == NULL) { file_oomem(ms); return NULL; } mfn is a local variable and is never free'd after successful pass through this function. Actually, it looks like mfn is *never* used in this function, so it might be possible to eliminate this whole block using the included patch. --- apprentice.c.orig 2003-10-16 19:07:43.000000000 +0000 +++ apprentice.c 2003-10-16 19:09:17.000000000 +0000 @@ -215,7 +215,7 @@ protected struct mlist * file_apprentice(struct magic_set *ms, const char *fn, int action) { - char *p, *mfn, *afn = NULL; + char *p, *afn = NULL; int file_err, errs = -1; struct mlist *mlist; @@ -224,13 +224,7 @@ if (fn == NULL) fn = MAGIC; - if ((fn = mfn = strdup(fn)) == NULL) { - file_oomem(ms); - return NULL; - } - if ((mlist = malloc(sizeof(*mlist))) == NULL) { - free(mfn); file_oomem(ms); return NULL; } @@ -244,7 +238,6 @@ break; if (ms->flags & MAGIC_MIME) { if ((afn = malloc(strlen(fn) + 5 + 1)) == NULL) { - free(mfn); free(mlist); file_oomem(ms); return NULL; @@ -263,7 +256,6 @@ fn = p; } if (errs == -1) { - free(mfn); free(mlist); mlist = NULL; file_error(ms, 0, "could not find any magic files!"); brett From christos at zoulas.com Thu Oct 16 15:28:32 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: Memory leak in apprentice.c In-Reply-To: <3104.192.168.123.158.1066331380.squirrel@staff.deepfile.com> from "Brett Funderburg" (Oct 16, 2:09pm) Message-ID: <20031016192832.428E97E03@beowulf.gw.com> On Oct 16, 2:09pm, brettf@deepfile.com ("Brett Funderburg") wrote: -- Subject: Memory leak in apprentice.c | Using valgrind --leak-check=yes on file-4.06, it reports that | apprentice.c line 227 is leaking memory. | It appears in this bit of code: | | if ((fn = mfn = strdup(fn)) == NULL) { | file_oomem(ms); | return NULL; | } | | mfn is a local variable and is never free'd after successful pass through | this function. Actually, it looks like mfn is *never* used in this | function, so it might be possible to eliminate this whole block using the | included patch. The code is really tricky there. It uses fn, which becomes an alias to mfn after the strdup(). You need to do that because fn is const char * and will get trashed by the function (and if indeed it points to const storage [text] it will core dump). The right fix is to put a free(mfn) just before the last return. christos From brettf at deepfile.com Tue Dec 23 11:16:38 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: release status? Message-ID: <3000.192.168.123.158.1072196198.squirrel@staff.deepfile.com> Christos, do you plan to cut a 4.07 release any time soon which will include the update to the python bindings to support the new magic_errno api you added in the 4.06 release? I previously sent you the updated files in e-mail. Let me know if you need me to repost them. Thanks, brett From christos at zoulas.com Tue Dec 23 12:58:07 2003 From: christos at zoulas.com (Christos Zoulas) Date: Sat Mar 5 00:36:57 2005 Subject: file 4.07 is now available Message-ID: <20031223175807.2E4687E05@beowulf.gw.com> >From ftp.astron.com:/pub/file/file-4.07.tar.gz This version fixes: Fix -k flag Fix large file support. Be more lenient on corrupted elf files Updated python bindings. Fix off by one error on input strings Add MAGIC_ERROR and magic_errno() Lots of magic fixes/additions Enjoy, christos From brettf at deepfile.com Tue Dec 23 13:23:45 2003 From: brettf at deepfile.com (Brett Funderburg) Date: Sat Mar 5 00:36:57 2005 Subject: file 4.07 is now available In-Reply-To: <20031223175807.2E4687E05@beowulf.gw.com> References: <20031223175807.2E4687E05@beowulf.gw.com> Message-ID: <3476.192.168.123.158.1072203825.squirrel@staff.deepfile.com> That's what I call service :) brett > >>From ftp.astron.com:/pub/file/file-4.07.tar.gz > > This version fixes: > > Fix -k flag > Fix large file support. > Be more lenient on corrupted elf files > Updated python bindings. > Fix off by one error on input strings > Add MAGIC_ERROR and magic_errno() > Lots of magic fixes/additions > > Enjoy, > > christos > _______________________________________________ > File mailing list > File@mx.gw.com > http://mx.gw.com/mailman/listinfo/file >