From oscaruser at programmer.net  Wed Jan 20 02:30:30 2010
From: oscaruser at programmer.net (oscaruser at programmer.net)
Date: Tue, 19 Jan 2010 19:30:30 -0500
Subject: Crystal Reports Mime Type Designation
Message-ID: <8CC677EE78F3FA6-E00-38DA@web-mmc-d13.sysops.aol.com>


 Greetings Christos,

I've been using mime-type designation, 'application/x-rpt; charset=binary' for Crystal Reports based on FileExt.com's registry. http://filext.com/file-extension/RPC. Is there an official answer? Is this type be handled in the code at all? I would like to leave this file identification up to the source maintainer rather than keep a branch update. I have recently inquired with SAP.com (10 minutes ago) to see if they can (or will) answer this question in some kind of official manner, but I suspect my request is /dev/null'd.

Thank you,
-OSC  :D
 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100119/e22b1a5d/attachment.html>

From christos at zoulas.com  Wed Jan 20 03:13:51 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 19 Jan 2010 20:13:51 -0500
Subject: Crystal Reports Mime Type Designation
In-Reply-To: <8CC677EE78F3FA6-E00-38DA@web-mmc-d13.sysops.aol.com>
	from oscaruser@programmer.net (Jan 19,  7:30pm)
Message-ID: <20100120011351.8D9E95654E@rebar.astron.com>

On Jan 19,  7:30pm, oscaruser at programmer.net (oscaruser at programmer.net) wrote:
-- Subject: Crystal Reports Mime Type Designation

|  Greetings Christos,
| 
| I've been using mime-type designation, 'application/x-rpt; charset=3Dbinary=
| ' for Crystal Reports based on FileExt.com's registry. http://filext.com/fi=
| le-extension/RPC. Is there an official answer? Is this type be handled in t=
| he code at all? I would like to leave this file identification up to the so=
| urce maintainer rather than keep a branch update. I have recently inquired =
| with SAP.com (10 minutes ago) to see if they can (or will) answer this ques=
| tion in some kind of official manner, but I suspect my request is /dev/null=
| 'd.
| 

I guess that's ok, and we can make file output that.

christos


From christos at zoulas.com  Fri Jan 22 23:51:13 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Fri, 22 Jan 2010 16:51:13 -0500
Subject: file-5.04 is now available
Message-ID: <20100122215113.AF1EE5654E@rebar.astron.com>

Hello,

This is a bug fix and security release. There are no new features.

	ftp://ftp.astron.com/pub/file/file-5.04.tar.gz

Enjoy,

christos

2010-01-22  15:45  Christos Zoulas <christos at zoulas.com>

	* print proper mime for crystal reports file

	* print the last summary information of a cdf document, not the
	  first so that nested documents print the right info

2010-01-16  18:42  Charles Longeau <chl at tuxfamily.org>

	* bring back some fixes from OpenBSD:
		- make gcc2 builds file
		- fix typos in a magic file comment

2009-11-17  18:35  Christos Zoulas <christos at zoulas.com>

	* ctime/asctime can return NULL on some OS's although
	  they should not (Toshit Antani)

2009-09-14  13:49  Christos Zoulas <christos at zoulas.com>

	* Centralize magic path handling routines and remove the
	  special-casing from file.c so that the python module for
	  example comes up with the same magic path (Fixes ~/.magic
	  handling) (from Gab)

2009-09-11  23:38  Reuben Thomas <rrt at sc3d.org>

	* When magic argument is a directory, read the files in
	  strcmp-sorted order (fixes Debian bug #488562 and our own FIXME).

2009-09-11  13:11  Reuben Thomas <rrt at sc3d.org>

	* Combine overlapping epoc and psion magic files into one (epoc).

	* Add some more EPOC MIME types.

2009-08-19  15:55  Christos Zoulas <christos at zoulas.com>

	* Fix 3 bugs (From Ian Darwin):
	    - file_showstr could move one past the end of the array
	    - parse_apple did not nul terminate the string in the overflow case
	    - parse_mime truncated the wrong string in the overflow case

2009-08-12  12:28  Robert Byrnes  <byrnes at wildpumpkin.net>

	* Include Localstuff when compiling magic.

2009-07-15  10:05  Christos Zoulas <christos at zoulas.com>

	* Fix logic for including mygetopts.h

	* Make cdf.c compile again with debugging

	* Add the necessary field handling for crystal reports files to work

2009-06-23 01:34  Reuben Thomas <rrt at sc3d.org>

	* Stop "(if" identifying Lisp files, that's plain dumb!

2009-06-09 22:13  Reuben Thomas <rrt at sc3d.org>

	* Add a couple of missing MP3 MIME types.

2009-05-27 23:00  Reuben Thomas <rrt at sc3d.org>

	* Add full range of hash-bang tests for Python and Ruby.

	* Add MIME types for Python and Ruby scripts.

2009-05-13  10:44  Christos Zoulas <christos at zoulas.com>

	* off by one in parsing hw capabilities in elf
	  (Cheng Renquan)

2009-05-08  13:40  Christos Zoulas <christos at zoulas.com>
	
	* lint fixes and more from NetBSD

2009-05-06  10:25  Christos Zoulas <christos at zoulas.com>

	* Avoid null dereference in cdf code (Drew Yao)

	* More cdf bounds checks and overflow checks


From gillen.daniel at gmail.com  Thu Jan 28 10:22:06 2010
From: gillen.daniel at gmail.com (Gillen Daniel)
Date: Thu, 28 Jan 2010 09:22:06 +0100
Subject: Question regarding MAGIC_NO_CHECK_* flags
Message-ID: <4B61492E.6020907@gmail.com>

Hi @all

I'm currently developing an application for browsing file systems and 
want to implement libmagic in order to get the type of displayed files. 
As I don't need a human readable output, but one that I can use 
internally, I created my own magic file. During tests, I saw that 
libmagic is doing some internal stuff to recognize some file types 
(ASCII files for ex.).

I found some MAGIC_NO_CHECK_* defines in the header file which I can use 
to disable this behavior. But porting my app from an older library 
version to 5.03 (Ubuntu 9.04 -> Ubuntu 9.10), I have to use more flags 
to disable all the internal checks.

My question now is if there is some sort of special flag that disables 
all internal checks whatever version of libmagic I'm using just leaving 
the file recognition to the magic file.

Thx in advance

Dan



From christos at zoulas.com  Thu Jan 28 17:06:59 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 28 Jan 2010 10:06:59 -0500
Subject: Question regarding MAGIC_NO_CHECK_* flags
In-Reply-To: <4B61492E.6020907@gmail.com> from Gillen Daniel (Jan 28,  9:22am)
Message-ID: <20100128150659.C69C95654E@rebar.astron.com>

On Jan 28,  9:22am, gillen.daniel at gmail.com (Gillen Daniel) wrote:
-- Subject: Question regarding MAGIC_NO_CHECK_* flags

| Hi @all
| 
| I'm currently developing an application for browsing file systems and 
| want to implement libmagic in order to get the type of displayed files. 
| As I don't need a human readable output, but one that I can use 
| internally, I created my own magic file. During tests, I saw that 
| libmagic is doing some internal stuff to recognize some file types 
| (ASCII files for ex.).
| 
| I found some MAGIC_NO_CHECK_* defines in the header file which I can use 
| to disable this behavior. But porting my app from an older library 
| version to 5.03 (Ubuntu 9.04 -> Ubuntu 9.10), I have to use more flags 
| to disable all the internal checks.
| 
| My question now is if there is some sort of special flag that disables 
| all internal checks whatever version of libmagic I'm using just leaving 
| the file recognition to the magic file.

Well,

1. These flags are going to change from version to version because of
   the addition/removal of features.
2. One of the flags is to disable the magic file itself, which is just
   an additional check, so what you really want is NO_CHECK_ALL, but
   check SOFTMAGIC. I don't think it is unreasonable though to add
   NO_CHECK_BUILTIN to do what you want. It will not help you with older
   versions of the library.
3. Some formats cannot be detected with magic files (microsoft documents,
   tar files, etc.) so disabling those checks cripples functionality.
4. If you want standardized output why don't you use mime?

christos


From dnovotny at redhat.com  Thu Jan 28 16:52:47 2010
From: dnovotny at redhat.com (Daniel Novotny)
Date: Thu, 28 Jan 2010 09:52:47 -0500 (EST)
Subject: segfault in star.ulaw
In-Reply-To: <1132987823.305851264690204138.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
Message-ID: <2081858899.306171264690367028.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>

hello,

our user has filed a Fedora bug about file segfault:

$ file star.ulaw 
Segmentation fault (core dumped)

now I was able to analyze the error and propose a fix:
the file is a raw sound sample, I have uploaded it to

http://people.fedoraproject.org/~dnovotny/f/star.ulaw

a gdb analysis of the crash shows line 950 in softmagic.c

for (lines = linecnt, b = buf; lines &&
                             ((b = CAST(const char *,
                                 memchr(c = b, '\n', CAST(size_t, (end - b)))))
                             || (b = CAST(const char *,
                                 memchr(c, '\r', CAST(size_t, (end - c))))));
                             lines--, b++) {
                                last = b;
                                if (b[0] == '\r' && b[1] == '\n')
                                        b++;
                        }
for some reason the pointer "b" have run outside of the array and b > end,
so the substraction yields negative number, which is then cast to unsigned

my patch adds "b < end && c < end" to the "for" condition,
so the cycle will not run away from "our" portion of memory

this seems to fix the bug (the file is then classified as "data"),
I tried a few sanity tests with the patched program and everything
seems to work normally

I attach backtrace and the patch, I can provide a core file, too,
but the bug is easily reproducible with the test file

downstream bug report link for reference:
https://bugzilla.redhat.com/show_bug.cgi?id=533245

regards,

   Daniel Novotny, Red Hat inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-ulaw-segfault.patch
Type: text/x-patch
Size: 553 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100128/88d767c4/attachment.bin>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: file-ulaw-segfault-backtrace.txt
URL: <http://mx.gw.com/pipermail/file/attachments/20100128/88d767c4/attachment.txt>

From gillen.daniel at gmail.com  Thu Jan 28 20:41:41 2010
From: gillen.daniel at gmail.com (Gillen Daniel)
Date: Thu, 28 Jan 2010 19:41:41 +0100
Subject: Question regarding MAGIC_NO_CHECK_* flags
In-Reply-To: <4B61D5C9.1010405@gmail.com>
References: <20100128150659.C69C95654E@rebar.astron.com>
	<4B61D5C9.1010405@gmail.com>
Message-ID: <4B61DA65.1000602@gmail.com>

Hi again

I'm really sorry but I just realized that I used my stripped magic file
to do the tests :(

Using the shipped magic file, I can differentiate between .tar.gz and
.png files using mime output and I am able to group all images with
"image/*" for example. This should do the trick for me.

Thx anyway for your help.

Dan

Gillen Daniel wrote:
> Hi
>
> Thx for your answer.
>
> A flag like NO_CHECK_BUILTIN would be great. Nevertheless this won't fix
> my issue with older libraries, it would fix it at least in the future :)
>
> Everything that isn't supported by the magic file could be done manually
> afterwards by my app. This wouldn't be a problem for me as I only need a
> small amount of files to be identified. The other could be classified
> more generally (binary, text etc...).
>
> I did look at the mime output, but this is already too specific / not
> specific enough for me. My app is handing files over to analyzer plugins
> depending on the file type. And there I don't need to know whether a
> file is for example a c++ text file or a plain ASCII file. I just need
> to know if it is plain ASCII or not. While this would be possible with
> mime types by looking at the first part before the "/", it doesn't work
> for example to differentiate between a .tar.gz and a .png file, as both
> produce application/octed-stream and I have to call a different plugin
> for .tar.gz than for .png files.
>
> Dan
>
> Christos Zoulas wrote:
>   
>> On Jan 28,  9:22am, gillen.daniel at gmail.com (Gillen Daniel) wrote:
>> -- Subject: Question regarding MAGIC_NO_CHECK_* flags
>>
>> | Hi @all
>> | 
>> | I'm currently developing an application for browsing file systems and 
>> | want to implement libmagic in order to get the type of displayed files. 
>> | As I don't need a human readable output, but one that I can use 
>> | internally, I created my own magic file. During tests, I saw that 
>> | libmagic is doing some internal stuff to recognize some file types 
>> | (ASCII files for ex.).
>> | 
>> | I found some MAGIC_NO_CHECK_* defines in the header file which I can use 
>> | to disable this behavior. But porting my app from an older library 
>> | version to 5.03 (Ubuntu 9.04 -> Ubuntu 9.10), I have to use more flags 
>> | to disable all the internal checks.
>> | 
>> | My question now is if there is some sort of special flag that disables 
>> | all internal checks whatever version of libmagic I'm using just leaving 
>> | the file recognition to the magic file.
>>
>> Well,
>>
>> 1. These flags are going to change from version to version because of
>>    the addition/removal of features.
>> 2. One of the flags is to disable the magic file itself, which is just
>>    an additional check, so what you really want is NO_CHECK_ALL, but
>>    check SOFTMAGIC. I don't think it is unreasonable though to add
>>    NO_CHECK_BUILTIN to do what you want. It will not help you with older
>>    versions of the library.
>> 3. Some formats cannot be detected with magic files (microsoft documents,
>>    tar files, etc.) so disabling those checks cripples functionality.
>> 4. If you want standardized output why don't you use mime?
>>
>> christos
>>
>> _______________________________________________
>> File mailing list
>> File at mx.gw.com
>> http://mx.gw.com/mailman/listinfo/file
>>   
>>     
>
>   


From gillen.daniel at gmail.com  Thu Jan 28 20:22:01 2010
From: gillen.daniel at gmail.com (Gillen Daniel)
Date: Thu, 28 Jan 2010 19:22:01 +0100
Subject: Question regarding MAGIC_NO_CHECK_* flags
In-Reply-To: <20100128150659.C69C95654E@rebar.astron.com>
References: <20100128150659.C69C95654E@rebar.astron.com>
Message-ID: <4B61D5C9.1010405@gmail.com>

Hi

Thx for your answer.

A flag like NO_CHECK_BUILTIN would be great. Nevertheless this won't fix
my issue with older libraries, it would fix it at least in the future :)

Everything that isn't supported by the magic file could be done manually
afterwards by my app. This wouldn't be a problem for me as I only need a
small amount of files to be identified. The other could be classified
more generally (binary, text etc...).

I did look at the mime output, but this is already too specific / not
specific enough for me. My app is handing files over to analyzer plugins
depending on the file type. And there I don't need to know whether a
file is for example a c++ text file or a plain ASCII file. I just need
to know if it is plain ASCII or not. While this would be possible with
mime types by looking at the first part before the "/", it doesn't work
for example to differentiate between a .tar.gz and a .png file, as both
produce application/octed-stream and I have to call a different plugin
for .tar.gz than for .png files.

Dan

Christos Zoulas wrote:
> On Jan 28,  9:22am, gillen.daniel at gmail.com (Gillen Daniel) wrote:
> -- Subject: Question regarding MAGIC_NO_CHECK_* flags
>
> | Hi @all
> | 
> | I'm currently developing an application for browsing file systems and 
> | want to implement libmagic in order to get the type of displayed files. 
> | As I don't need a human readable output, but one that I can use 
> | internally, I created my own magic file. During tests, I saw that 
> | libmagic is doing some internal stuff to recognize some file types 
> | (ASCII files for ex.).
> | 
> | I found some MAGIC_NO_CHECK_* defines in the header file which I can use 
> | to disable this behavior. But porting my app from an older library 
> | version to 5.03 (Ubuntu 9.04 -> Ubuntu 9.10), I have to use more flags 
> | to disable all the internal checks.
> | 
> | My question now is if there is some sort of special flag that disables 
> | all internal checks whatever version of libmagic I'm using just leaving 
> | the file recognition to the magic file.
>
> Well,
>
> 1. These flags are going to change from version to version because of
>    the addition/removal of features.
> 2. One of the flags is to disable the magic file itself, which is just
>    an additional check, so what you really want is NO_CHECK_ALL, but
>    check SOFTMAGIC. I don't think it is unreasonable though to add
>    NO_CHECK_BUILTIN to do what you want. It will not help you with older
>    versions of the library.
> 3. Some formats cannot be detected with magic files (microsoft documents,
>    tar files, etc.) so disabling those checks cripples functionality.
> 4. If you want standardized output why don't you use mime?
>
> christos
>
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file
>   


From daniel.leidert.spam at gmx.net  Fri Jan 29 15:42:38 2010
From: daniel.leidert.spam at gmx.net (Daniel Leidert)
Date: Fri, 29 Jan 2010 14:42:38 +0100
Subject: Detection of application/xslt+xml (XSLT)
Message-ID: <1264772558.6243.50.camel@haktar.wgdd.de>

Hi,

When I try to get the MIME type of an XSLT file I receive
application/xml. This is probably due to the magic file. I would suggest
to return application/xslt+xml if xsl:stylesheet or xsl:transform is
found. What do you think? Sample file attached.

Regards, Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.xsl
Type: application/xslt+xml
Size: 193 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100129/accefe82/attachment.bin>

From schwehr at ccom.unh.edu  Mon Feb  1 20:45:29 2010
From: schwehr at ccom.unh.edu (Kurt Schwehr)
Date: Mon, 01 Feb 2010 13:45:29 -0500
Subject: Marine sciences surveying data formats
Message-ID: <4B672149.8030109@ccom.unh.edu>

Hi Christos and company,

I've been working up file definitions for common data types in marine 
geology/geophysics and oceanography.  Many of the formats that I've been 
working with do not have very good identifiable characteristics, but I 
figured that after a year, it time so submit at least a bigging batch of 
files.  The complete file that I have been working on is here:

http://vislab-ccom.unh.edu/~schwehr/software/simplesegy/magic

Attached are the definitions that are worthy of consideration for 
inclusion in file.  Please let me know what you think.

Thanks!
-kurt
Res. Assist. Professor, Ocean Engineering, UNH CCOM/JHC
http://schwehr.org/blog

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: magic-submit
URL: <http://mx.gw.com/pipermail/file/attachments/20100201/2d568b89/attachment.ksh>

From christos at zoulas.com  Mon Feb  1 23:39:19 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 1 Feb 2010 16:39:19 -0500
Subject: Marine sciences surveying data formats
In-Reply-To: <4B672149.8030109@ccom.unh.edu>
	from Kurt Schwehr (Feb  1,  1:45pm)
Message-ID: <20100201213919.446AD5654F@rebar.astron.com>

On Feb 1,  1:45pm, schwehr at ccom.unh.edu (Kurt Schwehr) wrote:
-- Subject: Marine sciences surveying data formats

| Hi Christos and company,
| 
| I've been working up file definitions for common data types in marine 
| geology/geophysics and oceanography.  Many of the formats that I've been 
| working with do not have very good identifiable characteristics, but I 
| figured that after a year, it time so submit at least a bigging batch of 
| files.  The complete file that I have been working on is here:
| 
| http://vislab-ccom.unh.edu/~schwehr/software/simplesegy/magic
| 
| Attached are the definitions that are worthy of consideration for 
| inclusion in file.  Please let me know what you think.
| 
| Thanks!
| -kurt
| Res. Assist. Professor, Ocean Engineering, UNH CCOM/JHC
| http://schwehr.org/blog

Thanks Kurt,

For the most part these look ok. I am a bit worried about the <= 4 bytes
of magic ones, but we'll try them and see how they work out.

christos


From ghelmer at palisadesys.com  Sat Feb  6 00:48:53 2010
From: ghelmer at palisadesys.com (Guy Helmer)
Date: Fri, 5 Feb 2010 16:48:53 -0600
Subject: file-5.04 and Excel 2007 files
Message-ID: <E6DB43A1-75D9-42DF-AAA8-DA7FA13B1E1E@palisadesys.com>

I'm trying to identity Excel 2007 files; I thought file 5.04 would be able to, but I am still getting the result:
Book1.xlsx: Zip archive data, at least v2.0 to extract

Any hints?

Thanks,
Guy Helmer

From christos at zoulas.com  Sat Feb  6 01:32:17 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Fri, 5 Feb 2010 18:32:17 -0500
Subject: file-5.04 and Excel 2007 files
In-Reply-To: <E6DB43A1-75D9-42DF-AAA8-DA7FA13B1E1E@palisadesys.com>
	from Guy Helmer (Feb  5,  4:48pm)
Message-ID: <20100205233217.6E5A45654E@rebar.astron.com>

On Feb 5,  4:48pm, ghelmer at palisadesys.com (Guy Helmer) wrote:
-- Subject: file-5.04 and Excel 2007 files

| I'm trying to identity Excel 2007 files; I thought file 5.04 would be able to, but I am still getting the result:
| Book1.xlsx: Zip archive data, at least v2.0 to extract

not yet. I need to link in a libzip so we can look inside.

christos


From schwehr at ccom.unh.edu  Sun Feb  7 14:39:02 2010
From: schwehr at ccom.unh.edu (Kurt Schwehr)
Date: Sun, 07 Feb 2010 07:39:02 -0500
Subject: Marine sciences surveying data formats
In-Reply-To: <20100201213919.446AD5654F@rebar.astron.com>
References: <20100201213919.446AD5654F@rebar.astron.com>
Message-ID: <4B6EB466.8070801@ccom.unh.edu>

Sounds good.  If some of these don't work, I understand.  I threw out a 
whole bunch before submitting them already.  Just let me know which ones 
cause trouble or work and I will flag them in my development file.

Thanks!
-kurt

Christos Zoulas wrote:
> | Hi Christos and company,
> | 
> | I've been working up file definitions for common data types in marine 
> | geology/geophysics and oceanography.  Many of the formats that I've been 
> | working with do not have very good identifiable characteristics, but I 
> | figured that after a year, it time so submit at least a bigging batch of 
> | files.  The complete file that I have been working on is here:
> | 
> | http://vislab-ccom.unh.edu/~schwehr/software/simplesegy/magic
> | 
> | Attached are the definitions that are worthy of consideration for 
> | inclusion in file.  Please let me know what you think.
> | 
>
> Thanks Kurt,
>
> For the most part these look ok. I am a bit worried about the <= 4 bytes
> of magic ones, but we'll try them and see how they work out.
>
> christos
>
>   


From christos at zoulas.com  Sun Feb  7 18:58:40 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 7 Feb 2010 11:58:40 -0500
Subject: Marine sciences surveying data formats
In-Reply-To: <4B6EB466.8070801@ccom.unh.edu>
	from Kurt Schwehr (Feb  7,  7:39am)
Message-ID: <20100207165840.CA0BF5654E@rebar.astron.com>

On Feb 7,  7:39am, schwehr at ccom.unh.edu (Kurt Schwehr) wrote:
-- Subject: Re: Marine sciences surveying data formats

| Sounds good.  If some of these don't work, I understand.  I threw out a 
| whole bunch before submitting them already.  Just let me know which ones 
| cause trouble or work and I will flag them in my development file.
| 

I put them all in for now. We'll see which ones stick :-)

Thanks,

christos


From oscaruser at programmer.net  Tue Feb  9 02:10:11 2010
From: oscaruser at programmer.net (oscaruser at programmer.net)
Date: Mon, 08 Feb 2010 19:10:11 -0500
Subject: file-5.04 and Excel 2007 files
In-Reply-To: <20100205233217.6E5A45654E@rebar.astron.com>
Message-ID: <8CC7733616FFDAF-1B0-2ADD@web-mmc-m06.sysops.aol.com>

This is what we devised for the 'OpenOffice' doc recognition. It is a 
front end perl script, and returns the mime type, e.g. file -i option.  
It accepts either a stream input or filename passed from the command 
line. Assumes file util installed at /usr/local/bin/file. 
Comments/updates welcome.

OSC


-----Original Message-----
From: Christos Zoulas <christos at zoulas.com>
To: File Utility <file at mx.gw.com>
Sent: Fri, Feb 5, 2010 3:32 pm
Subject: Re: file-5.04 and Excel 2007 files


On Feb 5,  4:48pm, ghelmer at palisadesys.com (Guy Helmer) wrote:
-- Subject: file-5.04 and Excel 2007 files

| I'm trying to identity Excel 2007 files; I thought file 5.04 would be 
able to,
but I am still getting the result:
| Book1.xlsx: Zip archive data, at least v2.0 to extract

not yet. I need to link in a libzip so we can look inside.

christos

_______________________________________________
File mailing list
File at mx.gw.com
http://mx.gw.com/mailman/listinfo/file

  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mime_type.pl
Type: application/octet-stream
Size: 4272 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100208/4614a2c1/attachment.obj>

From daxim at cpan.org  Fri Feb 19 19:30:55 2010
From: daxim at cpan.org (Lars =?utf-8?b?RMmq4bSH4bSE4bSL4bSP4bShIOi/quaLieaWrw==?=)
Date: Fri, 19 Feb 2010 18:30:55 +0100
Subject: POD files with =encoding
Message-ID: <201002191831.08068.daxim@cpan.org>

Hello list,

attached is a patch that makes file recognise POD files that start with the 
=encoding command paragraph. It applies cleanly against 5.04.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-http-perldoc.perl.org-perlpod.html-3dencoding-_encod.patch
Type: text/x-patch
Size: 912 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100219/55d6c733/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mx.gw.com/pipermail/file/attachments/20100219/55d6c733/attachment-0001.bin>

From j at hug.gs  Wed Feb 24 00:19:44 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 23 Feb 2010 14:19:44 -0800
Subject: [patch] Man page fixes
Message-ID: <fe8323e11002231419p28d830du5b5627a1569ea8bc@mail.gmail.com>

The "message" column somehow lost its .It macro.  While I was fixing that I
noticed some other errors and fixed those as well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100223/5a89a00e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-man.patch
Type: application/octet-stream
Size: 2177 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100223/5a89a00e/attachment.obj>

From j at hug.gs  Wed Feb 24 00:18:05 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 23 Feb 2010 14:18:05 -0800
Subject: [patch] Windows PE fixes
Message-ID: <fe8323e11002231418o48faf7bdyc770900b3bc1066a@mail.gmail.com>

These attached patches are an attempt to clean up the msdos module.  Among
other things, Windows CE binaries and system images are properly recognized
now, and x64 handling works even in the presence of sections containing
MSIL.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100223/958d2528/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-msdos.patch
Type: application/octet-stream
Size: 13410 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100223/958d2528/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-lynx-omf-disambiguate.patch
Type: application/octet-stream
Size: 658 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100223/958d2528/attachment-0001.obj>

From christos at zoulas.com  Wed Feb 24 01:29:32 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 23 Feb 2010 18:29:32 -0500
Subject: [patch] Man page fixes
In-Reply-To: <fe8323e11002231419p28d830du5b5627a1569ea8bc@mail.gmail.com>
	from "Dr. Jesus" (Feb 23,  2:19pm)
Message-ID: <20100223232932.8970B5654E@rebar.astron.com>

On Feb 23,  2:19pm, j at hug.gs ("Dr. Jesus") wrote:
-- Subject: [patch] Man page fixes

Thanks a lot!

christos


From christos at zoulas.com  Wed Feb 24 01:37:19 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 23 Feb 2010 18:37:19 -0500
Subject: [patch] Windows PE fixes
In-Reply-To: <fe8323e11002231418o48faf7bdyc770900b3bc1066a@mail.gmail.com>
	from "Dr. Jesus" (Feb 23,  2:18pm)
Message-ID: <20100223233719.1AB015654E@rebar.astron.com>

On Feb 23,  2:18pm, j at hug.gs ("Dr. Jesus") wrote:
-- Subject: [patch] Windows PE fixes

| These attached patches are an attempt to clean up the msdos module.  Among
| other things, Windows CE binaries and system images are properly recognized
| now, and x64 handling works even in the presence of sections containing
| MSIL.

Applied, thanks.

christos


From j at hug.gs  Thu Mar 18 21:07:14 2010
From: j at hug.gs (Dr. Jesus)
Date: Thu, 18 Mar 2010 12:07:14 -0700
Subject: WIM file
Message-ID: <fe8323e11003181207s71bb4fd8tefe0dfc0cbc6edd8@mail.gmail.com>

For the msdos module.  Sorry, no patch, this is on top of the previous
patch I sent you and I guess the file project isn't under source
control.



# Windows Imaging (WIM) Image
0       string  MSWIM\000\000\000       Windows imaging (WIM) image


From christos at zoulas.com  Thu Mar 18 21:51:21 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 18 Mar 2010 15:51:21 -0400
Subject: WIM file
In-Reply-To: <fe8323e11003181207s71bb4fd8tefe0dfc0cbc6edd8@mail.gmail.com>
	from "Dr. Jesus" (Mar 18, 12:07pm)
Message-ID: <20100318195121.CC9E656425@rebar.astron.com>

On Mar 18, 12:07pm, j at hug.gs ("Dr. Jesus") wrote:
-- Subject: WIM file

| For the msdos module.  Sorry, no patch, this is on top of the previous
| patch I sent you and I guess the file project isn't under source
| control.
| 
| 
| 
| # Windows Imaging (WIM) Image
| 0       string  MSWIM\000\000\000       Windows imaging (WIM) image

Added, thanks.

christos


From Jens.Schleusener at t-systems-sfr.com  Fri Mar 19 12:11:58 2010
From: Jens.Schleusener at t-systems-sfr.com (Jens Schleusener)
Date: Fri, 19 Mar 2010 11:11:58 +0100 (CET)
Subject: Obvious "ASCII text" file detected as "HTML document text"
Message-ID: <alpine.LSU.2.01.1003191055100.15638@dino.rz.go.dlr.de>

Hi,

within the "SfR Fresh"-Archiv filetype detection is done with the 
great program "file".

For HTML files automatically a rendered display is tried ("simple" text 
representation is also available).

So I just detected a "file" problem, see (incorrectly interpreted as HTML)

  http://www.sfr-fresh.com/unix/www/eprints-3.2.0.tar.gz:a/eprints-3.2.0/NEWS

(the text file is incorrectly interpreted as HTML) and

  http://www.sfr-fresh.com/unix/www/eprints-3.2.0.tar.gz:t/eprints-3.2.0/NEWS

where the same file is correctly displayed as text.

Analyzing that file I got the impression that the only existence of the 
string

  <head>

anywhere in the analyzed otherwise pure text file and without any other 
"HTML"-indications caused that errorneous (?) behaviour.

In the mentioned example that is the line:

   * Added more metadata to the <head> of the summary pages, for the
     benefit of search engines.

Regards

Jens

-- 
T-Systems Solutions for Research GmbH
Solutions & Innovations Commercial ICT, Internet- & Intranet-Appl.
Dr. Jens Schleusener
Bunsenstr. 10, D-37073 G?ttingen
+49 551 709-2493 (Tel.)
+49 551 709-2169 (Fax)
E-Mail: Jens.Schleusener at t-systems.com
Internet: http://www.t-systems.com

T-Systems Solutions for Research GmbH
Management Board: J?rgen Aumayer (Chairman), Hans Gersing
Commercial register: Amtsgericht M?nchen, HRB 12 55 01,
Registered Office: We?ling
VAT ident no.: DE 193456493

Notice: This transmittal and/or attachments may be privileged or
confidential. If you are not the intended recipient, you are hereby
notified that you have received this transmittal in error; any review,
dissemination, or copying is strictly prohibited. If you received
this transmittal in error, please notify us immediately by reply and
immediately delete this message and all its attachments. Thank you.

From ian at darwinsys.com  Sat Mar 20 19:34:40 2010
From: ian at darwinsys.com (Ian Darwin)
Date: Sat, 20 Mar 2010 13:34:40 -0400
Subject: Obvious "ASCII text" file detected as "HTML document text"
In-Reply-To: <alpine.LSU.2.01.1003191055100.15638@dino.rz.go.dlr.de>
References: <alpine.LSU.2.01.1003191055100.15638@dino.rz.go.dlr.de>
Message-ID: <4BA50730.8010007@darwinsys.com>

Jens Schleusener wrote:
> Hi,
>
> within the "SfR Fresh"-Archiv filetype detection is done with the 
> great program "file".
>
> For HTML files automatically a rendered display is tried ("simple" 
> text representation is also available).
>
> So I just detected a "file" problem, see (incorrectly interpreted as 
> HTML)
> ...
> In the mentioned example that is the line:
>
>   * Added more metadata to the <head> of the summary pages, for the
>     benefit of search engines.

Thanks, but file(1) is not an AI program. I think you should not put 
HTML elements in
text files if it is important to you that they be displayed as plain 
text. This includes
putting HTML elements in CVS/SVN/... commit logs which get saved as 
text. There is
no reason the committer could not have written:

 > * Added more metadata to the "head" section of the summary pages, ...

Ian


From Jens.Schleusener at t-systems-sfr.com  Sun Mar 21 21:38:59 2010
From: Jens.Schleusener at t-systems-sfr.com (Jens Schleusener)
Date: Sun, 21 Mar 2010 20:38:59 +0100 (CET)
Subject: Obvious "ASCII text" file detected as "HTML document text"
In-Reply-To: <4BA50730.8010007@darwinsys.com>
References: <alpine.LSU.2.01.1003191055100.15638@dino.rz.go.dlr.de>
	<4BA50730.8010007@darwinsys.com>
Message-ID: <alpine.LSU.2.01.1003211955400.17984@dino.rz.go.dlr.de>

Hi Ian,

>> within the "SfR Fresh"-Archiv filetype detection is done with the
>> great program "file".
>>
>> For HTML files automatically a rendered display is tried ("simple"
>> text representation is also available).
>>
>> So I just detected a "file" problem, see (incorrectly interpreted as
>> HTML)
>> ...
>> In the mentioned example that is the line:
>>
>>   * Added more metadata to the <head> of the summary pages, for the
>>     benefit of search engines.
>
> Thanks, but file(1) is not an AI program. I think you should not put 
> HTML elements in text files if it is important to you that they be 
> displayed as plain text.

Thanks for your answer but

1) I am not the author of the just as an example mentioned file (with the 
incorrect detected file type) but the maintainer of the mentioned software 
archive.

2) although "file" isn't an AI program detecting pure HTML files shouldn't 
be too difficult, see http://www.w3.org/TR/html401/struct/global.html:

  An HTML 4 document is composed of three parts:

  1. a line containing HTML version information,
  2. a declarative header section (delimited by the HEAD element),
  3. a body, which contains the document's actual content. The body may
     be implemented by the BODY element or the FRAMESET element.

  White space (spaces, newlines, tabs, and comments) may appear before or
  after each section. Sections 2 and 3 should be delimited by the HTML
  element. Here's an example of a simple HTML document:

  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
     "http://www.w3.org/TR/html4/strict.dtd">
  <HTML>
     <HEAD>
        <TITLE>My first HTML document</TITLE>
     </HEAD>
     <BODY>
        <P>Hello world!
     </BODY>
  </HTML>

Ok, a problem may be non-HTML filed containing HTML code snippets,

> This includes putting HTML elements in CVS/SVN/... commit logs which get 
> saved as text. There is no reason the committer could not have written:
>
> > * Added more metadata to the "head" section of the summary pages, ...

That's right.

But after doing some more testing to breakdown the problem I have the 
impression that is a problem introduced in the current release 5.0.4:

Using the following test file "file_html_test.txt"

000000000000000000000000000000000000000000000000000000
11111 This is obviously not a HTML document text 11111
222222222222222222222222222222222222222222222222222222
333333333333333333333333<head>333333333333333333333333
444444444444444444444444444444444444444444444444444444
555555555 But "file" unfortunately think so! 555555555
666666666666666666666666666666666666666666666666666666

I got the following results with self-compiled binaries
(under OpenSUSE 11,2):

a) file (release 5.0.3):

  file_html_test.txt: ASCII text

b) file (release 5.0.4):

  file_html_test.txt: HTML document text

But in the new Changelog entries of release 5.0.4 I couldn't find an entry 
regarding that different behaviour. Sorry, I am not a "file" expert.
Any debugging hints to find the matching magic rules?

Regards

Jens


From ian at darwinsys.com  Sun Mar 21 23:52:56 2010
From: ian at darwinsys.com (Ian Darwin)
Date: Sun, 21 Mar 2010 17:52:56 -0400
Subject: Obvious "ASCII text" file detected as "HTML document text"
In-Reply-To: <alpine.LSU.2.01.1003211955400.17984@dino.rz.go.dlr.de>
References: <alpine.LSU.2.01.1003191055100.15638@dino.rz.go.dlr.de>	<4BA50730.8010007@darwinsys.com>
	<alpine.LSU.2.01.1003211955400.17984@dino.rz.go.dlr.de>
Message-ID: <4BA69538.1080700@darwinsys.com>

Jens Schleusener wrote:
> Hi Ian,
>
>>> within the "SfR Fresh"-Archiv filetype detection is done with the
>>> great program "file".
>>>
>>> For HTML files automatically a rendered display is tried ("simple"
>>> text representation is also available).
>>>
>>> So I just detected a "file" problem, see (incorrectly interpreted as
>>> HTML)
>>> ...
>>> In the mentioned example that is the line:
>>>
>>>   * Added more metadata to the <head> of the summary pages, for the
>>>     benefit of search engines.
>>
>> Thanks, but file(1) is not an AI program. I think you should not put 
>> HTML elements in text files if it is important to you that they be 
>> displayed as plain text.
>
> Thanks for your answer but
>
> 1) I am not the author of the just as an example mentioned file (with 
> the incorrect detected file type) but the maintainer of the mentioned 
> software archive.
>
> 2) although "file" isn't an AI program detecting pure HTML files 
> shouldn't be too difficult, see 
> http://www.w3.org/TR/html401/struct/global.html:
>
>  An HTML 4 document is composed of three parts:
>
>  1. a line containing HTML version information,
>  2. a declarative header section (delimited by the HEAD element),
>  3. a body, which contains the document's actual content. The body may
>     be implemented by the BODY element or the FRAMESET element.
>
>  White space (spaces, newlines, tabs, and comments) may appear before or
>  after each section. Sections 2 and 3 should be delimited by the HTML
>  element. Here's an example of a simple HTML document:
>
Yes, I'm quite familiar with the HTML document format, having been the 
person
who released the world's first commercial HTML editor back in the early 
1990's.

So. You want "file" to build in an XML/XHTML/HTML4 parser, so we can handle
DOCTYPEs, xmlns's, comments, spaces? Not gonna happen. Maybe a regex?

\s?(<[^>]+>)+

But it will fail if the user has a valid HTML file like
<html><head><title>This is not a tag</title>

I don't know what the answer is, but it has to be something that doesn't 
make file
even slower than it's already become.

I'm guessing we'll end up having to write another C module to handle
HTML, XML, XHTML and related stuff. Thoughts?


From christos at zoulas.com  Mon Mar 22 01:16:45 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 21 Mar 2010 19:16:45 -0400
Subject: Obvious "ASCII text" file detected as "HTML document text"
In-Reply-To: <4BA69538.1080700@darwinsys.com> from Ian Darwin (Mar 21,  5:52pm)
Message-ID: <20100321231645.217BE56425@rebar.astron.com>

On Mar 21,  5:52pm, ian at darwinsys.com (Ian Darwin) wrote:
-- Subject: Re: Obvious "ASCII text" file detected as "HTML document text"

| Yes, I'm quite familiar with the HTML document format, having been the 
| person
| who released the world's first commercial HTML editor back in the early 
| 1990's.
| 
| So. You want "file" to build in an XML/XHTML/HTML4 parser, so we can handle
| DOCTYPEs, xmlns's, comments, spaces? Not gonna happen. Maybe a regex?
| 
| \s?(<[^>]+>)+
| 
| But it will fail if the user has a valid HTML file like
| <html><head><title>This is not a tag</title>
| 
| I don't know what the answer is, but it has to be something that doesn't 
| make file
| even slower than it's already become.
| 
| I'm guessing we'll end up having to write another C module to handle
| HTML, XML, XHTML and related stuff. Thoughts?

The problem is that browser will eat anything that remotely resembles
html and try to render it. This has created a lot of pages that are not
standards conformant, but nevertheless are displayed correctly in browsers.

We went through approximately 50000 samples of web pages in the latest
version of file, and before the changes to the magic file we had a recognition
rate of approximately 60%; after the changes we have 99%. 

Sure we can write another c module for html, but let's come up with a spec
for it first and see if this cannot be implemented as magic.

christos


From dnovotny at redhat.com  Wed Mar 24 15:20:48 2010
From: dnovotny at redhat.com (Daniel Novotny)
Date: Wed, 24 Mar 2010 09:20:48 -0400 (EDT)
Subject: squashfs filesystem version 4.0: magic entry update
In-Reply-To: <2059648198.1302691269436836796.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
Message-ID: <1814248542.1302711269436848259.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>

hello,

one of our users discovered, that the magic entry for squashfs doesn't count
with changes in the new version 4.0. 

the superblock format is different now,
I had to add this data according to squashfs source code, 
patch against file 5.04 attached

downstream bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=550212

regards,

 Daniel Novotny, Red Hat inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-squashfs.patch
Type: text/x-patch
Size: 1516 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100324/20eaeb1e/attachment.bin>

From christos at zoulas.com  Wed Mar 24 16:17:58 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 24 Mar 2010 10:17:58 -0400
Subject: squashfs filesystem version 4.0: magic entry update
In-Reply-To: <1814248542.1302711269436848259.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
	from Daniel Novotny (Mar 24,  9:20am)
Message-ID: <20100324141758.33DA356426@rebar.astron.com>

On Mar 24,  9:20am, dnovotny at redhat.com (Daniel Novotny) wrote:
-- Subject: squashfs filesystem version 4.0: magic entry update

| hello,
| 
| one of our users discovered, that the magic entry for squashfs doesn't count
| with changes in the new version 4.0. 
| 
| the superblock format is different now,
| I had to add this data according to squashfs source code, 
| patch against file 5.04 attached
| 
| downstream bug report:
| https://bugzilla.redhat.com/show_bug.cgi?id=550212
| 
| regards,
| 
|  Daniel Novotny, Red Hat inc.

Thanks muchly!

christos


From dnovotny at redhat.com  Wed Apr 14 14:05:43 2010
From: dnovotny at redhat.com (Daniel Novotny)
Date: Wed, 14 Apr 2010 07:05:43 -0400 (EDT)
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <316557957.245871271242751652.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
Message-ID: <655364602.246131271243143464.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>

hello,

we found, that file returns zero exit code ("success") in case of "no such file or directory" error:

$ file nosuch
nosuch: cannot open `nosuch' (No such file or directory)
$ echo $?
0

the cause is in fsmagic.c:

ret = stat(fn, sb);     /* don't merge into if; see "ret =" above */

        if (ret) {
                if (ms->flags & MAGIC_ERROR) {
                        file_error(ms, errno, "cannot stat `%s'", fn);
                        return -1;
                }
                if (file_printf(ms, "cannot open `%s' (%s)",
                    fn, strerror(errno)) == -1)
                        return -1;
                return 1;
        }

the error is printed with file_printf and "return 1" means success in this case

changing this to "file_error" and "return -1" all the time
(not just when there is MAGIC_ERROR flag) causes this to work

$ file nosuch
nosuch: ERROR: cannot open `nosuch' (No such file or directory)
$ echo $?
1

patch against 5.04 attached

maybe there is some purpose, why the error condition is printed
with file_printf and no error returned, but I didn't find any
maybe you have some idea?

  regards,

     Daniel Novotny, Red Hat inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-retval.patch
Type: text/x-patch
Size: 650 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100414/bd149c33/attachment.bin>

From christos at zoulas.com  Wed Apr 14 16:21:31 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 14 Apr 2010 09:21:31 -0400
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <655364602.246131271243143464.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
	from Daniel Novotny (Apr 14,  7:05am)
Message-ID: <20100414132131.857F156425@rebar.astron.com>

On Apr 14,  7:05am, dnovotny at redhat.com (Daniel Novotny) wrote:
-- Subject: the file command returns zero exit code even in case of not exist

| hello,
| 
| we found, that file returns zero exit code ("success") in case of "no such file or directory" error:
| 
| $ file nosuch
| nosuch: cannot open `nosuch' (No such file or directory)
| $ echo $?
| 0
| 
| the cause is in fsmagic.c:
| 
| ret = stat(fn, sb);     /* don't merge into if; see "ret =" above */
| 
|         if (ret) {
|                 if (ms->flags & MAGIC_ERROR) {
|                         file_error(ms, errno, "cannot stat `%s'", fn);
|                         return -1;
|                 }
|                 if (file_printf(ms, "cannot open `%s' (%s)",
|                     fn, strerror(errno)) == -1)
|                         return -1;
|                 return 1;
|         }
| 
| the error is printed with file_printf and "return 1" means success in this case
| 
| changing this to "file_error" and "return -1" all the time
| (not just when there is MAGIC_ERROR flag) causes this to work
| 
| $ file nosuch
| nosuch: ERROR: cannot open `nosuch' (No such file or directory)
| $ echo $?
| 1
| 
| patch against 5.04 attached
| 
| maybe there is some purpose, why the error condition is printed
| with file_printf and no error returned, but I didn't find any
| maybe you have some idea?

Imagine the scenario where we have to classify many files. What should
happen if one of them does not exist/cannot be read?

According to: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html

    If file does not exist, cannot be read, or its file status could
    not be determined, the output shall indicate that the file was
    processed, but that its type could not be determined.

So it is reallly not an error if something went wrong dealing with a particular
file. "The output" means stdout to me.

christos


From dnovotny at redhat.com  Thu Apr 22 18:25:18 2010
From: dnovotny at redhat.com (Daniel Novotny)
Date: Thu, 22 Apr 2010 11:25:18 -0400 (EDT)
Subject: file may trim too much of command line from core file
In-Reply-To: <2122324832.773621271949743197.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
Message-ID: <1158966586.774031271949918377.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>

hello,

there is a problem with core file analysis:
the command line of the crashed program is sometimes trimmed in a way
the text is not complete and only later part of the string is shown:

$ file core.2493
core.2493: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from
'gee --rid=48373'

$ which gee
/usr/bin/which: no gee in
(/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/stephent/bin)

$ strings core.2493 | head | grep gee
/usr/bin/python /usr/lib/python2.6/site-packages/rpdb2.py --debugee --rid=48373

I have found a Debian bugzilla entry for this, with a patch attached
(the patch searches for other sections in the core file)
The same thing was reported in Fedora, I used the Debian patch
and it succesfully works both in our development as well as stable OS release

The patch is attached, most credits go to Arnaud Giersch <arnaud.giersch at iut-bm.univ-fcomte.fr>
from the Debian community

links to downstream bug reports follow -
Debian bugzilla: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=422524
Red Hat bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=566305

 best regards,

     Daniel Novotny, Red Hat inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-core-trim.patch
Type: text/x-patch
Size: 769 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100422/86d166c7/attachment.bin>

From christos at zoulas.com  Thu Apr 22 19:55:45 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 22 Apr 2010 12:55:45 -0400
Subject: file may trim too much of command line from core file
In-Reply-To: <1158966586.774031271949918377.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
	from Daniel Novotny (Apr 22, 11:25am)
Message-ID: <20100422165545.D042656425@rebar.astron.com>

On Apr 22, 11:25am, dnovotny at redhat.com (Daniel Novotny) wrote:
-- Subject: file may trim too much of command line from core file

| there is a problem with core file analysis:
| the command line of the crashed program is sometimes trimmed in a way
| the text is not complete and only later part of the string is shown:
| 
| $ file core.2493
| core.2493: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from
| 'gee --rid=48373'
| 
| $ which gee
| /usr/bin/which: no gee in
| (/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/stephent/bin)
| 
| $ strings core.2493 | head | grep gee
| /usr/bin/python /usr/lib/python2.6/site-packages/rpdb2.py --debugee --rid=48373
| 
| I have found a Debian bugzilla entry for this, with a patch attached
| (the patch searches for other sections in the core file)
| The same thing was reported in Fedora, I used the Debian patch
| and it succesfully works both in our development as well as stable OS release
| 
| The patch is attached, most credits go to Arnaud Giersch <arnaud.giersch at iut-bm.univ-fcomte.fr>
| from the Debian community
| 
| links to downstream bug reports follow -
| Debian bugzilla: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=422524
| Red Hat bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=566305
| 
|  best regards,
| 
|      Daniel Novotny, Red Hat inc.

got it, thanks a lot!

christos


From a.nielsen at shikadi.net  Sat Apr 24 16:56:15 2010
From: a.nielsen at shikadi.net (Adam Nielsen)
Date: Sat, 24 Apr 2010 23:56:15 +1000
Subject: Bug?  UPX .exe files not detected
Message-ID: <4BD2F87F.5010806@shikadi.net>

Hi all,

I've just discovered that while the latest version of file has an apparently
correct magic string for detecting UPX-compressed .exe files, for some reason
they never seem to be detected.

I've uploaded a 1k test file at http://www.shikadi.net/files/file-upxtest.dat
which contains enough data for the magic string to match, but all I get is a
message saying it's a standard Win32 .exe - no mention of UPX.

As far as I can tell the magic string is accurate, am I missing some option or
is this a bug in file?

Thanks,
Adam.


From christos at zoulas.com  Sat Apr 24 18:09:35 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sat, 24 Apr 2010 11:09:35 -0400
Subject: Bug?  UPX .exe files not detected
In-Reply-To: <4BD2F87F.5010806@shikadi.net> from Adam Nielsen (Apr 24, 11:56pm)
Message-ID: <20100424150935.42E3356425@rebar.astron.com>

On Apr 24, 11:56pm, a.nielsen at shikadi.net (Adam Nielsen) wrote:
-- Subject: Bug?  UPX .exe files not detected

| Hi all,
| 
| I've just discovered that while the latest version of file has an apparently
| correct magic string for detecting UPX-compressed .exe files, for some reason
| they never seem to be detected.
| 
| I've uploaded a 1k test file at http://www.shikadi.net/files/file-upxtest.dat
| which contains enough data for the magic string to match, but all I get is a
| message saying it's a standard Win32 .exe - no mention of UPX.
| 
| As far as I can tell the magic string is accurate, am I missing some option or
| is this a bug in file?

This is what the current version of file says for me on your data:

file-upxtest.dat: PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed

christos


From a.nielsen at shikadi.net  Sun Apr 25 03:14:04 2010
From: a.nielsen at shikadi.net (Adam Nielsen)
Date: Sun, 25 Apr 2010 10:14:04 +1000
Subject: Bug?  UPX .exe files not detected
In-Reply-To: <20100424150935.42E3356425@rebar.astron.com>
References: <20100424150935.42E3356425@rebar.astron.com>
Message-ID: <4BD3894C.2000703@shikadi.net>

> | As far as I can tell the magic string is accurate, am I missing some option or
> | is this a bug in file?
> 
> This is what the current version of file says for me on your data:
> 
> file-upxtest.dat: PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed

Thanks for your reply!  That's really strange, using file-5.04 this is what I get:

$ ./configure && make
$ cd src
$ export LD_LIBRARY_PATH=.libs/
$ strace ./file -m ../magic/magic.mgc ../file-upxtest.dat
...
open(".libs/libmagic.so.1", O_RDONLY)   = 3
read(3, "\177ELF\2\1\1\0"..., 832) = 832
...
open("../magic/magic.mgc", O_RDONLY)    = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1779600, ...}) = 0

../file-upxtest.dat: PE32 executable for MS Windows (GUI) Intel 80386 32-bit

As far as I can tell this is using the correct binary, correct library and
correct magic file.  Am I missing something?  The message is very obviously
different to yours, apparently in an older style too.  file-5.04 was the most
recent version I could find.

Thanks again,
Adam.


From christos at zoulas.com  Sun Apr 25 03:32:43 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sat, 24 Apr 2010 20:32:43 -0400
Subject: Bug?  UPX .exe files not detected
In-Reply-To: <4BD3894C.2000703@shikadi.net> from Adam Nielsen (Apr 25, 10:14am)
Message-ID: <20100425003243.1A79F56425@rebar.astron.com>

On Apr 25, 10:14am, a.nielsen at shikadi.net (Adam Nielsen) wrote:
-- Subject: Re: Bug?  UPX .exe files not detected

| > | As far as I can tell the magic string is accurate, am I missing some option or
| > | is this a bug in file?
| > 
| > This is what the current version of file says for me on your data:
| > 
| > file-upxtest.dat: PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows, UPX compressed
| 
| Thanks for your reply!  That's really strange, using file-5.04 this is what I get:
| 
| $ ./configure && make
| $ cd src
| $ export LD_LIBRARY_PATH=.libs/
| $ strace ./file -m ../magic/magic.mgc ../file-upxtest.dat
| ...
| open(".libs/libmagic.so.1", O_RDONLY)   = 3
| read(3, "\177ELF\2\1\1\0"..., 832) = 832
| ...
| open("../magic/magic.mgc", O_RDONLY)    = 3
| fstat(3, {st_mode=S_IFREG|0644, st_size=1779600, ...}) = 0
| 
| ../file-upxtest.dat: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
| 
| As far as I can tell this is using the correct binary, correct library and
| correct magic file.  Am I missing something?  The message is very obviously
| different to yours, apparently in an older style too.  file-5.04 was the most
| recent version I could find.

No, the reason mine works is because I am running stuff at the head of
the tree that has particular msdos fixes. I'll release 5.05 soon.

christos


From a.nielsen at shikadi.net  Sun Apr 25 03:43:43 2010
From: a.nielsen at shikadi.net (Adam Nielsen)
Date: Sun, 25 Apr 2010 10:43:43 +1000
Subject: Bug?  UPX .exe files not detected
In-Reply-To: <20100425003243.1A79F56425@rebar.astron.com>
References: <20100425003243.1A79F56425@rebar.astron.com>
Message-ID: <4BD3903F.6080901@shikadi.net>

> No, the reason mine works is because I am running stuff at the head of
> the tree that has particular msdos fixes. I'll release 5.05 soon.

Ah ok, that explains things, glad to know it's been fixed!  I look forward to
the next release.

Cheers,
Adam.


From tledouxfr at gmail.com  Wed May  5 20:28:02 2010
From: tledouxfr at gmail.com (Thomas Ledoux)
Date: Wed, 5 May 2010 19:28:02 +0200
Subject: Mimetypes missing
Message-ID: <v2v248062181005051028ibf8d3a60vb11427e954fd3677@mail.gmail.com>

Hello,

some mimetypes are missing :
- in the definition of the icc profile files
(as specified by the iana
http://www.iana.org/assignments/media-types/application/vnd.iccprofile)

- in some of the cases for mpeg sequences

The attached patch does add those in the Magdir/sun and Magdir/animation files

Thanks
    Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file5.04-mimetypes.patch
Type: application/octet-stream
Size: 1800 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100505/f301c1d5/attachment.obj>

From christos at zoulas.com  Wed May  5 20:38:33 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 5 May 2010 13:38:33 -0400
Subject: Mimetypes missing
In-Reply-To: <v2v248062181005051028ibf8d3a60vb11427e954fd3677@mail.gmail.com>
	from Thomas Ledoux (May  5,  7:28pm)
Message-ID: <20100505173833.8661156425@rebar.astron.com>

On May 5,  7:28pm, tledouxfr at gmail.com (Thomas Ledoux) wrote:
-- Subject: Mimetypes missing

| Hello,
| 
| some mimetypes are missing :
| - in the definition of the icc profile files
| (as specified by the iana
| http://www.iana.org/assignments/media-types/application/vnd.iccprofile)
| 
| - in some of the cases for mpeg sequences
| 
| The attached patch does add those in the Magdir/sun and Magdir/animation files
| 
| Thanks
|     Thomas

Thanks, added.

christos


From woodbrian77 at gmail.com  Mon May 31 00:38:45 2010
From: woodbrian77 at gmail.com (Brian Wood)
Date: Sun, 30 May 2010 16:38:45 -0500
Subject: File functionality in library format
Message-ID: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>

Greetings,

I'm working on something called the C++ Middleware Writer.  It's an on line
service that writes C++ output based on user input.  User input files are
uploaded to the site as part of the process.  I would like to be able
to call the
file code in a library form from my process rather than making a call to system
with file as an argument.  It would be great if I could get a version
of the code
that was adapted for that.    I've downloaded, built and installed the 5.04
version so have gotten that far.  I looked at 2010 and 2009 archives and
didn't find anything related to this.  I believe with the growth of on line
services there's a need for a library form of the functionality.

-- 
Brian Wood
Ebenezer Enterprises
http://webEbenezer.net
(651) 251-9384


From daxim at cpan.org  Mon May 31 14:51:55 2010
From: daxim at cpan.org (Lars =?utf-8?b?RMmq4bSH4bSE4bSL4bSP4bShIOi/quaLieaWrw==?=)
Date: Mon, 31 May 2010 13:51:55 +0200
Subject: File functionality in library format
In-Reply-To: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
References: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
Message-ID: <201005311352.05741.daxim@cpan.org>

man 3 libmagic

(That was easy.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mx.gw.com/pipermail/file/attachments/20100531/6d302fec/attachment.bin>

From sacha at ssl.co.uk  Mon May 31 15:16:45 2010
From: sacha at ssl.co.uk (Sacha Varma)
Date: Mon, 31 May 2010 13:16:45 +0100
Subject: File functionality in library format
In-Reply-To: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
References: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
Message-ID: <4C03A8AD.1080101@ssl.co.uk>

Brian Wood wrote:
> I would like to be able to call the
> file code in a library form from my process rather than making a call to system
> with file as an argument.  It would be great if I could get a version
> of the code that was adapted for that.

Hi Brian - we use the 'file' package like that by using the 'magic' 
library that it builds (libmagic.a/.so on UNIX). The calls you need are 
in magic.h - magic_load(), magic_file() etc.

Our experience has been that although you can use the package in this 
way, it seems to be really developed to support the 'file' command, and 
things can change a lot between releases (for example, we're stuck on 
4.19 for some fundamental reason that I can't now recall).



From christos at zoulas.com  Tue Jun  1 00:44:10 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 31 May 2010 17:44:10 -0400
Subject: File functionality in library format
In-Reply-To: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
	from Brian Wood (May 30,  4:38pm)
Message-ID: <20100531214410.EA3C656425@rebar.astron.com>

On May 30,  4:38pm, woodbrian77 at gmail.com (Brian Wood) wrote:
-- Subject: File functionality in library format

| Greetings,
| 
| I'm working on something called the C++ Middleware Writer.  It's an on line
| service that writes C++ output based on user input.  User input files are
| uploaded to the site as part of the process.  I would like to be able
| to call the
| file code in a library form from my process rather than making a call to system
| with file as an argument.  It would be great if I could get a version
| of the code
| that was adapted for that.    I've downloaded, built and installed the 5.04
| version so have gotten that far.  I looked at 2010 and 2009 archives and
| didn't find anything related to this.  I believe with the growth of on line
| services there's a need for a library form of the functionality.

After you install a recent version of file, look for libmagic, and
'man libmagic'.

christos


From woodbrian77 at gmail.com  Tue Jun  1 03:47:49 2010
From: woodbrian77 at gmail.com (Brian Wood)
Date: Mon, 31 May 2010 19:47:49 -0500
Subject: File functionality in library format
In-Reply-To: <20100531214410.EA3C656425@rebar.astron.com>
References: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
	<20100531214410.EA3C656425@rebar.astron.com>
Message-ID: <AANLkTimutlrzavxJcyCDBWI8jAnPigAcZf9ff-0gxaLl@mail.gmail.com>

On Mon, May 31, 2010 at 4:44 PM, Christos Zoulas <christos at zoulas.com> wrote:
> On May 30, ?4:38pm, woodbrian77 at gmail.com (Brian Wood) wrote:
> -- Subject: File functionality in library format
>
> | Greetings,
> |
> | I'm working on something called the C++ Middleware Writer. ?It's an on line
> | service that writes C++ output based on user input. ?User input files are
> | uploaded to the site as part of the process. ?I would like to be able
> | to call the
> | file code in a library form from my process rather than making a call to system
> | with file as an argument. ?It would be great if I could get a version
> | of the code
> | that was adapted for that. ? ?I've downloaded, built and installed the 5.04
> | version so have gotten that far. ?I looked at 2010 and 2009 archives and
> | didn't find anything related to this. ?I believe with the growth of on line
> | services there's a need for a library form of the functionality.
>
> After you install a recent version of file, look for libmagic, and
> 'man libmagic'.
>
> christos
>


Thanks.  It's working now.  I had some unresolved symbols
when I tried using libmagic.a but they went away when I
used the shared library.

-- 
Brian Wood
Ebenezer Enterprises
http://www.webEbenezer.net
(651) 251-9384


From woodbrian77 at gmail.com  Tue Jun  1 04:17:33 2010
From: woodbrian77 at gmail.com (Brian Wood)
Date: Mon, 31 May 2010 20:17:33 -0500
Subject: File functionality in library format
In-Reply-To: <201005311352.05741.daxim@cpan.org>
References: <AANLkTimmgbTjCz5vMEPaR0febXQiY_9NTr7n5kPENtqc@mail.gmail.com>
	<201005311352.05741.daxim@cpan.org>
Message-ID: <AANLkTikhEEbftQ4rR6PZ91H1DJnTcl6w38iq1EmJDZq0@mail.gmail.com>

On Mon, May 31, 2010 at 6:51 AM, Lars D?????? ??? <daxim at cpan.org> wrote:
> man 3 libmagic
>
> (That was easy.)
>
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file
>
>

Thanks.  I got it working.  When I tried linking with libmagic.a I
got some unresolved symbols, but when I used the shared library
they went away.

-- 
Brian Wood
Ebenezer Enterprises
http://www.webEbenezer.net
(651) 251-9384


From cchittleborough+comp.file at cluemail.com  Wed Jun  2 19:57:49 2010
From: cchittleborough+comp.file at cluemail.com (Christopher Chittleborough)
Date: Thu, 03 Jun 2010 02:27:49 +0930
Subject: Patch: minor markup fix in doc/magic.man
Message-ID: <1275497869.14885.1378132559@webmail.messagingengine.com>

Here's a patch to fix a small markup problem in doc/magic.man.

Cheers -- Chris

diff -rbu file-5.04.ORIGINAL/doc/magic.man file-5.04/doc/magic.man
--- file-5.04.ORIGINAL/doc/magic.man    2009-05-09 08:32:44.000000000
+0930
+++ file-5.04/doc/magic.man     2010-06-01 14:02:20.000000000 +0930
@@ -283,7 +283,7 @@
 The special test
 .Em x
 always evaluates to true.
-.Dv message
+.It Dv message
 The message to be printed if the comparison succeeds.
 If the string contains a
 .Xr printf 3


From christos at zoulas.com  Wed Jun  2 20:11:08 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 2 Jun 2010 13:11:08 -0400
Subject: Patch: minor markup fix in doc/magic.man
In-Reply-To: <1275497869.14885.1378132559@webmail.messagingengine.com>
	from "Christopher Chittleborough" (Jun  3,  2:27am)
Message-ID: <20100602171109.0440456425@rebar.astron.com>

On Jun 3,  2:27am, cchittleborough+comp.file at cluemail.com ("Christopher Chittleborough") wrote:
-- Subject: Patch: minor markup fix in doc/magic.man

| Here's a patch to fix a small markup problem in doc/magic.man.
| 
| Cheers -- Chris

thanks a lot!

christos


From jkaluza at redhat.com  Mon Jun 21 10:39:56 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Mon, 21 Jun 2010 03:39:56 -0400 (EDT)
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <148458309.502641277105630350.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
Message-ID: <2104358862.502931277105996509.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>

> | hello,
> | 
> | we found, that file returns zero exit code ("success") in case of "no such file or directory" error:
> | 
> | $ file nosuch
> | nosuch: cannot open `nosuch' (No such file or directory)
> | $ echo $?
> | 0
> | 
> | the cause is in fsmagic.c:
> | 
> | ret = stat(fn, sb);     /* don't merge into if; see "ret =" above */
> | 
> |         if (ret) {
> |                 if (ms->flags & MAGIC_ERROR) {
> |                         file_error(ms, errno, "cannot stat `%s'", fn);
> |                         return -1;
> |                 }
> |                 if (file_printf(ms, "cannot open `%s' (%s)",
> |                     fn, strerror(errno)) == -1)
> |                         return -1;
> |                 return 1;
> |         }
> | 
> | the error is printed with file_printf and "return 1" means success in this case
> | 
> | changing this to "file_error" and "return -1" all the time
> | (not just when there is MAGIC_ERROR flag) causes this to work
> | 
> | $ file nosuch
> | nosuch: ERROR: cannot open `nosuch' (No such file or directory)
> | $ echo $?
> | 1
> | 
> | patch against 5.04 attached
> | 
> | maybe there is some purpose, why the error condition is printed
> | with file_printf and no error returned, but I didn't find any
> | maybe you have some idea?
> 
> Imagine the scenario where we have to classify many files. What should
> happen if one of them does not exist/cannot be read?

I understand your situation and respect your decision, but I think more common behaviour is to return
an error code even one file from many doesn't exit, because that's what 
"ls", "cat", "chmod", "chown", "sort" and many others do. In my opinion file should
act also like those basic commands.

> According to: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html
> 
>     If file does not exist, cannot be read, or its file status could
>     not be determined, the output shall indicate that the file was
>     processed, but that its type could not be determined.

That's true, but I think we have to distinguish between the situation when file does not exist or cannot
be read and the situation when file status can't be determined.

I really don't want to somehow force you to do that, I've just wanted to bring some new ideas about this "problem".

> So it is reallly not an error if something went wrong dealing with a particular
> file. "The output" means stdout to me.
> 
> christos

Jan Kalu?a <jkaluza at redhat.com>


From christos at zoulas.com  Mon Jun 21 15:24:26 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 21 Jun 2010 08:24:26 -0400
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <2104358862.502931277105996509.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
	from Jan Kaluza (Jun 21,  3:39am)
Message-ID: <20100621122426.E597656425@rebar.astron.com>

On Jun 21,  3:39am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: the file command returns zero exit code even in case of not exist
| 
| > Imagine the scenario where we have to classify many files. What should
| > happen if one of them does not exist/cannot be read?
| 
| I understand your situation and respect your decision, but I think more common behaviour is to return
| an error code even one file from many doesn't exit, because that's what 
| "ls", "cat", "chmod", "chown", "sort" and many others do. In my opinion file should
| act also like those basic commands.
| 
| > According to: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html
| > 
| >     If file does not exist, cannot be read, or its file status could
| >     not be determined, the output shall indicate that the file was
| >     processed, but that its type could not be determined.
| 
| That's true, but I think we have to distinguish between the situation when file does not exist or cannot
| be read and the situation when file status can't be determined.
| 
| I really don't want to somehow force you to do that, I've just wanted to bring some new ideas about this "problem".
| 
| > So it is reallly not an error if something went wrong dealing with a particular
| > file. "The output" means stdout to me.
| > 
| > christos
| 
| Jan Kalu??a <jkaluza at redhat.com>

I did not say I disagree with you, in that it would make more sense for
file(1) should exit with non-zero if there was an error condition, but
an unreadable file or a broken symlink is not an error according to the
posix standard, and changing the behavior will not only violate posix,
but break existing scripts.

christos


From christos at zoulas.com  Mon Jun 21 15:25:34 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 21 Jun 2010 08:25:34 -0400
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <2104358862.502931277105996509.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>
	from Jan Kaluza (Jun 21,  3:39am)
Message-ID: <20100621122534.ADEAC56427@rebar.astron.com>

On Jun 21,  3:39am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: the file command returns zero exit code even in case of not exist


We could add an extra flag though to exit with non zero on a broken
symlink or an unreadable file. The question again is what to do with
multiple files? Keep going or exit on the first bad one?

christos


From jkaluza at redhat.com  Thu Jun 24 17:40:37 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Thu, 24 Jun 2010 10:40:37 -0400 (EDT)
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <20100621122426.E597656425@rebar.astron.com>
Message-ID: <249122402.808861277390437902.JavaMail.root@zmail04.collab.prod.int.phx2.redhat.com>



Jan Kalu?a <jkaluza at redhat.com>
----- "Christos Zoulas" <christos at zoulas.com> wrote:

> On Jun 21,  3:39am, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: the file command returns zero exit code even in case of
> not exist
> | 
> | > Imagine the scenario where we have to classify many files. What
> should
> | > happen if one of them does not exist/cannot be read?
> | 
> | I understand your situation and respect your decision, but I think
> more common behaviour is to return
> | an error code even one file from many doesn't exit, because that's
> what 
> | "ls", "cat", "chmod", "chown", "sort" and many others do. In my
> opinion file should
> | act also like those basic commands.
> | 
> | > According to:
> http://www.opengroup.org/onlinepubs/009695399/utilities/file.html
> | > 
> | >     If file does not exist, cannot be read, or its file status
> could
> | >     not be determined, the output shall indicate that the file
> was
> | >     processed, but that its type could not be determined.
> | 
> | That's true, but I think we have to distinguish between the
> situation when file does not exist or cannot
> | be read and the situation when file status can't be determined.
> | 
> | I really don't want to somehow force you to do that, I've just
> wanted to bring some new ideas about this "problem".
> | 
> | > So it is reallly not an error if something went wrong dealing with
> a particular
> | > file. "The output" means stdout to me.
> | > 
> | > christos
> | 
> | Jan Kalu?a <jkaluza at redhat.com>
> 
> I did not say I disagree with you, in that it would make more sense
> for
> file(1) should exit with non-zero if there was an error condition,
> but
> an unreadable file or a broken symlink is not an error according to
> the
> posix standard, and changing the behavior will not only violate
> posix,
> but break existing scripts.

I can't find those statements in posix standard just now,
so I will believe you (although I don't understand why some
basic commands return error code in those cases).
I also agree this change would break existing scripts.

As you have already mentioned in another email, it would
be fine to introduce new option which could return an error
code when file can't be read.

In my opinion, there are two ways how to handle multiple
files. The first one is to return error code even if one file from many
can't be read (I would say I prefer this one). The second one is to return
error code just when all files can't be read.

When thinking about it, maybe we can return an error code only when file(1)
handles just one file and keep the current behaviour when handling more files,
because error code when handling more files is really discussable.

> christos
> 
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Jan kaluza


From guy at alum.mit.edu  Thu Jun 24 20:09:05 2010
From: guy at alum.mit.edu (Guy Harris)
Date: Thu, 24 Jun 2010 10:09:05 -0700
Subject: the file command returns zero exit code even in case of not
	existing file being tested
In-Reply-To: <20100414132131.857F156425@rebar.astron.com>
References: <20100414132131.857F156425@rebar.astron.com>
Message-ID: <9177AC3D-8DE4-4F13-B8EA-C057EE654B97@alum.mit.edu>


On Apr 14, 2010, at 6:21 AM, Christos Zoulas wrote:

> According to: http://www.opengroup.org/onlinepubs/009695399/utilities/file.html
> 
>    If file does not exist, cannot be read, or its file status could
>    not be determined, the output shall indicate that the file was
>    processed, but that its type could not be determined.

And according to the 2008 (newer) version:

	http://www.opengroup.org/onlinepubs/9699919799/utilities/file.html

"If the file named by the <file> operand does not exist, cannot be read, or the type of the file named by the <file> operand cannot be determined, this shall not be considered an error that affects the exit status."

which is even more explicit.

So, yes, we'd need to add a new command-line flag if we want to be POSIX-compliant but also want to offer the option of exiting with an error code if a file doesn't exist or can't be read.

(As for history: if this is, indeed, the V7 file command:

	http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/file.c

and this is, indeed, the V7 C startup code:

	http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/libc/csu/crt0.s

then, if file had a reliable exit code in the "normal" case, it was through pure luck, as

	1) main() was just falling off the end

and

	2) (the lower 8 bits of) the return value of main ended up being the exit status

so the exit status was whatever happened to be in R0 at the time main() returned.  (It also had a "-f" flag, which read file names from the file name given as an argument to the flag, and reported on those files; if that failed, it exited with 2, but if it *succeeded*, it exited with 1.))


From jkaluza at redhat.com  Tue Jun 29 11:47:10 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 29 Jun 2010 10:47:10 +0200
Subject: Z-machine magic entry update
Message-ID: <201006291047.11272.jkaluza@redhat.com>

Hi,

file(1) sometimes identifies some files (.mp3 or Mono debug files - .mdb) as 
"Infocom game data". Most of those cases can be fixed by checking version of Z-
machine file (first byte), which is always between 1 and 8 [1] [2] [3]. I've 
created simple patch against file-5.04 which do that.

You can find the patch attached to this downstream bug:
https://bugzilla.redhat.com/show_bug.cgi?id=608922

[1] http://www.gnelson.demon.co.uk/zspec/sect11.html - original standard with 
version 1-6.
[2] http://www.jczorkmid.net/~jpenney/ZSpec11-latest.txt - latest standard 
with version 7,8.
[3] http://en.wikipedia.org/wiki/Z-machine - mention about version 7,8.

regards,
Jan Kaluza <jkaluza at redhat.com>


From christos at zoulas.com  Tue Jun 29 15:42:47 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 29 Jun 2010 08:42:47 -0400
Subject: Z-machine magic entry update
In-Reply-To: <201006291047.11272.jkaluza@redhat.com>
	from Jan Kaluza (Jun 29, 10:47am)
Message-ID: <20100629124247.4EE0256425@rebar.astron.com>

On Jun 29, 10:47am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Z-machine magic entry update

| Hi,
| 
| file(1) sometimes identifies some files (.mp3 or Mono debug files - .mdb) as 
| "Infocom game data". Most of those cases can be fixed by checking version of Z-
| machine file (first byte), which is always between 1 and 8 [1] [2] [3]. I've 
| created simple patch against file-5.04 which do that.
| 
| You can find the patch attached to this downstream bug:
| https://bugzilla.redhat.com/show_bug.cgi?id=608922
| 
| [1] http://www.gnelson.demon.co.uk/zspec/sect11.html - original standard with 
| version 1-6.
| [2] http://www.jczorkmid.net/~jpenney/ZSpec11-latest.txt - latest standard 
| with version 7,8.
| [3] http://en.wikipedia.org/wiki/Z-machine - mention about version 7,8.
| 

Thanks a lot; fixed as suggested.

christos


From jkaluza at redhat.com  Wed Jul  7 12:05:17 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Wed, 7 Jul 2010 11:05:17 +0200
Subject: WebM magic file
Message-ID: <201007071105.17877.jkaluza@redhat.com>

Hi

I'm attaching very basic magic file for WebM video format 
(http://www.webmproject.org/). Maybe we can somehow merge that with matroska, 
because they both use the same EBML header. It would be really great to have 
it in next file(1) release.

Thanks for your work
Jan Kaluza
-------------- next part --------------

#------------------------------------------------------------------------------
# $File: matroska,v 1.5 2009/09/27 19:02:12 christos Exp $
# webm:  file(1) magic for WebM files
#
# See http://www.webmproject.org/
#

# EBML id:
0		belong		0x1a45dfa3
# DocType id:
>0		search/4096 	\x42\x82
# DocType contents:
>>&1		string		webm	WebM
!:mime  video/webm

From lkundrak at v3.sk  Thu Jul  8 00:48:36 2010
From: lkundrak at v3.sk (Lubomir Rintel)
Date: Wed,  7 Jul 2010 17:48:36 -0400
Subject: [PATCH] Add matches for Parrot
Message-ID: <1278539316-2067-1-git-send-email-lkundrak@v3.sk>

Parrot is a virtual machine for highly portable byte code, primarily
generated by Perl 6 compilers such as Rakudo.
---
 magic/Magdir/parrot |   22 ++++++++++++++++++++++
 magic/Makefile.am   |    1 +
 2 files changed, 23 insertions(+), 0 deletions(-)
 create mode 100644 magic/Magdir/parrot

diff --git a/magic/Magdir/parrot b/magic/Magdir/parrot
new file mode 100644
index 0000000..400b311
--- /dev/null
+++ b/magic/Magdir/parrot
@@ -0,0 +1,22 @@
+#------------------------------------------------------------------------------
+# $File$
+# parrot: file(1) magic for Parrot Virtual Machine
+# URL:	http://www.lua.org/
+# From: Lubomir Rintel <lkundrak at v3.sk>
+
+# Compiled Parrot byte code
+0	string	\376PBC\r\n\032\n	Parrot bytecode
+>64	byte	x			%d.
+>72	byte	x			\b%d,
+>8	byte	>0			%d byte words,
+>16	byte	0			little-endian,
+>16	byte	1			big-endian,
+>32	byte	0			IEEE-754 8 byte double floats,
+>32	byte	1			x86 12 byte long double floats,
+>32	byte	2			IEEE-754 16 byte long double floats,
+>32	byte	3			MIPS 16 byte long double floats,
+>32	byte	4			AIX 16 byte long double floats,
+>32	byte	5			4-byte floats,
+>40	byte	x			Parrot %d.
+>48	byte	x			\b%d.
+>56	byte	x			\b%d
diff --git a/magic/Makefile.am b/magic/Makefile.am
index e10499d..53b01ae 100644
--- a/magic/Makefile.am
+++ b/magic/Makefile.am
@@ -149,6 +149,7 @@ $(MAGIC_FRAGMENT_DIR)/os9 \
 $(MAGIC_FRAGMENT_DIR)/osf1 \
 $(MAGIC_FRAGMENT_DIR)/palm \
 $(MAGIC_FRAGMENT_DIR)/parix \
+$(MAGIC_FRAGMENT_DIR)/parrot \
 $(MAGIC_FRAGMENT_DIR)/pbm \
 $(MAGIC_FRAGMENT_DIR)/pdf \
 $(MAGIC_FRAGMENT_DIR)/pdp \
-- 
1.6.5.2


From lkundrak at v3.sk  Thu Jul  8 00:49:35 2010
From: lkundrak at v3.sk (Lubomir Rintel)
Date: Wed,  7 Jul 2010 17:49:35 -0400
Subject: [PATCH] Add matches for ruby modules
Message-ID: <1278539375-2187-1-git-send-email-lkundrak@v3.sk>

Similar to what's already done for Perl. Existing rules only match shebangs,
which is not useful in most cases. This was tested to yield no false
positives when run against every perl and python source file I could find
on my system (260 CPAN modules and 59 Python modules). Produces a couple
of false negatives for a couple of files from 30 Ruby gems I have
installed, but is still an improvement over existing rules. (Ruby is
used to construct DSLs quite often, it would be really tricky to match
those).
---
 magic/Magdir/ruby |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/magic/Magdir/ruby b/magic/Magdir/ruby
index 7030295..be1786c 100644
--- a/magic/Magdir/ruby
+++ b/magic/Magdir/ruby
@@ -14,3 +14,15 @@
 !:mime text/x-ruby
 0	search/1	#!\ /usr/bin/env\ ruby	Ruby script text executable
 !:mime text/x-ruby
+
+# What looks like ruby, but does not have a shebang
+# (modules and such)
+# From: Lubomir Rintel <lkundrak at v3.sk>
+0	regex		\^[\ \t]*require[\ \t]'[A-Za-z_\/]+'
+>0	regex		include\ [A-Z]|def\ [a-z]|\ do$
+>>0	regex		\^[\ \t]*end([\ \t]*[;#].*)?$		Ruby script text
+!:mime	text/x-ruby
+0	regex		\^[\ \t]*(class|module)[\ \t][A-Z]
+>0	regex		(modul|includ)e\ [A-Z]|def\ [a-z]
+>>0	regex		\^[\ \t]*end([\ \t]*[;#].*)?$		Ruby module source text
+!:mime	text/x-ruby
-- 
1.6.5.2


From christos at zoulas.com  Thu Jul  8 23:18:59 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 8 Jul 2010 16:18:59 -0400
Subject: [PATCH] Add matches for Parrot
In-Reply-To: <1278539316-2067-1-git-send-email-lkundrak@v3.sk>
	from Lubomir Rintel (Jul  7,  5:48pm)
Message-ID: <20100708201859.C14F156425@rebar.astron.com>

On Jul 7,  5:48pm, lkundrak at v3.sk (Lubomir Rintel) wrote:
-- Subject: [PATCH] Add matches for Parrot

| Parrot is a virtual machine for highly portable byte code, primarily
| generated by Perl 6 compilers such as Rakudo.

Added, thanks!

christos


From swnykimo at gmail.com  Mon Jul 12 08:23:25 2010
From: swnykimo at gmail.com (Guo Lu)
Date: Mon, 12 Jul 2010 13:23:25 +0800
Subject: cross compile question
Message-ID: <AANLkTiljgBmsgICXGOhmjfsYzzO4bTTpzApf_gc5eWsQ@mail.gmail.com>

Hi,

My name is chen, and I am tried to cross compile
the "file" tool for arm based machine. I use CodeSourcery
cross compiler.
The compile commands I used are:

1: ./configure --host=arm-none-linux-gnueabi
2: make

there is no problem for configuring to generate the Makefile,
but when I execute the 'make' command, I got something
about magic number error:

file: Unknown !: entry `!:apple    8BIMGIFf'
make[2]: *** [magic.mgc] Error 255
make[2]: Leaving directory `/home/chen/tmp/file-5.00/magic'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/chen/tmp/file-5.00'
make: *** [all] Error 2

Can anybody give me any hints to solve this.
I'll appreciate very much

thanks
chen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100712/b0dbe294/attachment.html>

From quel at quelrod.net  Wed Jul 14 02:07:57 2010
From: quel at quelrod.net (James Nobis)
Date: Tue, 13 Jul 2010 18:07:57 -0500
Subject: .xlsx misidentified as a zip file
Message-ID: <4C3CF1CD.3080201@quelrod.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Ok it actually *is* a zip file that contains the Microsoft Excel 2007 XML format.

unzip -x test.xlsx
Archive:  test.xlsx
  inflating: xl/workbook.xml
  inflating: xl/worksheets/sheet1.xml
  inflating: xl/worksheets/sheet2.xml
  inflating: xl/worksheets/sheet3.xml
  inflating: xl/styles.xml
  inflating: xl/_rels/workbook.xml.rels
  inflating: [Content_Types].xml
  inflating: _rels/.rels

James
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQIcBAEBCgAGBQJMPPHNAAoJEGUWgJyjXssuOJYP/RRQBB+nOf2WR+O4qSVxZzl6
KTXRwusu3pWZlIHXakY7NUB1XS58Qe9LJBjfuL2Bv4OI8CByD9nN7+3JSW7tfHB5
ZYUKg9t9aSgnSM+ocdWfSKuPBgdG4wj6M6N5rTf03VCMOe0nxZ3vmJi9bGm7CndD
B115B/uwij0FHcu4B4M1zdVk/BwLFQcW3/JSasuaZ8Jy2vkmcwmW5LbWwLw5ucMA
cOa98s14JBPqRBz7EJ1jos8MlGIZ1G2CB6zr94gE/+meuHOiqlByDYFiD3xaHI17
Q1p8Y3gAA4t4YEFJJ4gdZSliucXEGNN/pZ8optobwsEYFdDi9pzGV2q7kONEoUuq
p9ZD6qwF9EUnbYMA744wc57k2iO/8dYP79wQKAfEEmWf7L7Vt+/2JWwp3q9ziikC
yZqNQ6pm7WJ4ZL3VCJMqH3033CQcnidgqF3zovcIwzKtBG/qls+yyQSIoyBaN+9+
xCJ9umS22HCJt5msaVmpFZLcwlYvhwWUYhGzw7a1Lc7sAakknNZdfa8zRqm8Pmij
hdAZAQD/6Soe7Pu/crNWLx1Qx/DKgG3uCoThWM7ho/IVd84nFPLvL6ikVzilL8aQ
yIivAg/c9T03dI587qvmgL+2OWyrJStTiTqTy5XRTrwSKRU8n3QiiYuQdCYCTFQo
lHtSYWQHg4kYDCsPNuUJ
=xpR4
-----END PGP SIGNATURE-----


From quel at quelrod.net  Wed Jul 14 02:00:15 2010
From: quel at quelrod.net (James Nobis)
Date: Tue, 13 Jul 2010 18:00:15 -0500
Subject: .xlsx misidentified as a zip file
Message-ID: <4C3CEFFF.3080206@quelrod.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

The .xlsx is Microsoft Excel 2007.  I'm using the 5.04 release and it gives me this: Zip archive
data, at least v2.0 to extract.

The first 4 bytes of the file:
50 4b 03 04
 P  K

I believe this matches the archive magic entry: 0  string      PK\003\004

I'm attaching a test file that gets this incorrect identification.  I created the file with Gnumeric
(Open Office doesn't appear to let you save in this format.)

James
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iQIcBAEBCgAGBQJMPO/+AAoJEGUWgJyjXssu9MgP/ixj9dmhYBYJcUm4M06SXi6K
rbQdzt8m6sesCnEekR6ljUSj81nI6G3yUMY2cgdeD+w08AHDJHq+tDTnkeVEykm7
Qq9ysgLrBfa/rUIAz9Ez2Fhkx0B3ZZ6aSV4hKhsqA/c3JDUBpkBeBH7Zp6KBXR8a
/dYEFCUaXVp5z/HMSagkWlJMtau86T/CycJ04mVgdF89VggcwczfiJbsa3DDodj/
Ue+26cTdHeJS4xmtcbYF9EI1yLVjd7cDmCGPCdmxbSyd+ozqAG27lUGGiOVMREV5
1+m2WhcVA65icdM0iLJpafxUoa7U8BYZNdMZb9la8KdHurxW6+PZ1Tgwn3JGgxYr
rGdZY0wgkFg5wH0K6RqHFV6HXp8Nz2jkw/5kKeK4L+Go5AyWA/0LY5T8u18s97bx
BKQp7ktEemrBLDhv45bpAnq8/E0VAu2iTBWeYG6Bm3xtybUFWwXVWPDtYaiIEaBD
QJtpG0sd82jbu9BT3qoB0jCPjUS0bIQxXRXJdB2UU+PJhMUTaV2sV3WygLmdqkap
0lyyDNXaRtI0GTQfIYj9ERP9nGpDyb6kfzZEz+3jgAxjtMieXQ7eQFLhJ0/aOkWE
iac2aWQcjbS/oHRI8p7n2eqL78H3J6/K/Zg4NbCbxu1PLCHqFCaLnzcqF1H685Bm
L1RnUedJfIj8kTdKfxlo
=a+aH
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.xlsx
Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Size: 3459 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100713/f96f7185/attachment.bin>

From christos at zoulas.com  Wed Jul 14 15:32:55 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 14 Jul 2010 08:32:55 -0400
Subject: .xlsx misidentified as a zip file
In-Reply-To: <4C3CF1CD.3080201@quelrod.net> from James Nobis (Jul 13,  6:07pm)
Message-ID: <20100714123255.A6B5856425@rebar.astron.com>

On Jul 13,  6:07pm, quel at quelrod.net (James Nobis) wrote:
-- Subject: .xlsx misidentified as a zip file

| Ok it actually *is* a zip file that contains the Microsoft Excel 2007 XML format.
| 
| unzip -x test.xlsx
| Archive:  test.xlsx
|   inflating: xl/workbook.xml
|   inflating: xl/worksheets/sheet1.xml
|   inflating: xl/worksheets/sheet2.xml
|   inflating: xl/worksheets/sheet3.xml
|   inflating: xl/styles.xml
|   inflating: xl/_rels/workbook.xml.rels
|   inflating: [Content_Types].xml
|   inflating: _rels/.rels

I will add support for Office 2007 zipped xml files later this year.

christos


From jkaluza at redhat.com  Mon Jul 19 12:30:46 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Mon, 19 Jul 2010 11:30:46 +0200
Subject: [PATCH] Fixed bad "from '%s'" for ELF some binaries
Message-ID: <201007191130.46391.jkaluza@redhat.com>

Hi,

attached patch fixes the bug when bad "from %s" string was printed for ELF 
binary. The problem is that "from %s" string is retrieved from "note section" 
from note with id 3 (NT_PRPSINFO). The function "donote" in readelf.c, which 
tries to retrieve that information, is shared between all types of ELF 
binaries (ET_CORE, ET_EXEC, ...). But for ET_EXEC, the note with id 3 is 
NT_GNU_BUILD_ID. Therefore file(1) returns bad "from %s" string if the binary 
is ET_EXEC type and contains NT_GNU_BUILD_ID "info".

My solution just adds one condition to detect if handled ELF file is ET_CORE 
and if it's not, it doesn't try to handle that note at all.

Downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=599695

If I can ask you, please confirm if you accept my patches, because as package 
maintainer I don't want to use patches which are not accepted by you. It would 
help me a lot.

Thanks for your work
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-core-prpsinfo.patch
Type: text/x-patch
Size: 904 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100719/d3c112fc/attachment.bin>

From lrn1986 at gmail.com  Sun Jul 18 17:32:41 2010
From: lrn1986 at gmail.com (LRN)
Date: Sun, 18 Jul 2010 18:32:41 +0400
Subject: Mingw compatibility patch
Message-ID: <4C431089.7000704@gmail.com>

  Directory diff against file 5.04 is attached.
It is not perfect by any measure, especially the bits in configure.ac 
and Makefile.am, and i've chosen to cull away missing functionality 
instead of porting it, such as symlink support (MSVCRT does not seem to 
be symlink-aware even if Windows is) and archive unpacking with 
fork()/exec() (can be rewritten with spawn(), but that's 
complicated)...but the code compiles, and file.exe works as intended 
(with altered magic.mgc search paths, yay!).

P.S. is there a way to prevent libtool from picking up compiled file.exe 
from the source tree when calling 'make' again? Because if this file.exe 
doesn't work (wich is often the case during development), libtool's 
magic tests will fail, and libtool will complain about missing real 
libraries.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-mingw.diff
Type: text/x-patch
Size: 12693 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100718/a5ed6da0/attachment.bin>

From christos at zoulas.com  Mon Jul 19 17:57:14 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 19 Jul 2010 10:57:14 -0400
Subject: [PATCH] Fixed bad "from '%s'" for ELF some binaries
In-Reply-To: <201007191130.46391.jkaluza@redhat.com>
	from Jan Kaluza (Jul 19, 11:30am)
Message-ID: <20100719145714.C1FA056426@rebar.astron.com>

On Jul 19, 11:30am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: [PATCH] Fixed bad "from '%s'" for ELF some binaries

| Hi,
| 
| attached patch fixes the bug when bad "from %s" string was printed for ELF 
| binary. The problem is that "from %s" string is retrieved from "note section" 
| from note with id 3 (NT_PRPSINFO). The function "donote" in readelf.c, which 
| tries to retrieve that information, is shared between all types of ELF 
| binaries (ET_CORE, ET_EXEC, ...). But for ET_EXEC, the note with id 3 is 
| NT_GNU_BUILD_ID. Therefore file(1) returns bad "from %s" string if the binary 
| is ET_EXEC type and contains NT_GNU_BUILD_ID "info".
| 
| My solution just adds one condition to detect if handled ELF file is ET_CORE 
| and if it's not, it doesn't try to handle that note at all.
| 
| Downstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=599695
| 
| If I can ask you, please confirm if you accept my patches, because as package 
| maintainer I don't want to use patches which are not accepted by you. It would 
| help me a lot.
| 
| Thanks for your work
| Jan Kaluza

Thanks for the patch, applied to HEAD.

Best,

christos


From christos at zoulas.com  Mon Jul 19 20:09:12 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 19 Jul 2010 13:09:12 -0400
Subject: Mingw compatibility patch
In-Reply-To: <4C431089.7000704@gmail.com>
       from LRN (Jul 18,  6:32pm)
Message-ID: <20100719170912.6767D56425@rebar.astron.com>

On Jul 18,  6:32pm, lrn1986 at gmail.com (LRN) wrote:
| Content-Transfer-Encoding: 7bit
| 
|   Directory diff against file 5.04 is attached.
| It is not perfect by any measure, especially the bits in configure.ac 
| and Makefile.am, and i've chosen to cull away missing functionality 
| instead of porting it, such as symlink support (MSVCRT does not seem to 
| be symlink-aware even if Windows is) and archive unpacking with 
| fork()/exec() (can be rewritten with spawn(), but that's 

I'd rather keep this around as a diff since I would prefer mingw32 to
fix their own issues (printf c99 standard formats etc), rather than
polluting the code.

| (with altered magic.mgc search paths, yay!).
| 
| P.S. is there a way to prevent libtool from picking up compiled file.exe 
| from the source tree when calling 'make' again? Because if this file.exe 
| doesn't work (wich is often the case during development), libtool's 
| magic tests will fail, and libtool will complain about missing real 
| libraries.

This is not desirable again because if the version of the magic file
changes with the new development code, building the magic file with
the old executable will not work.

christos


From lrn1986 at gmail.com  Tue Jul 20 08:23:03 2010
From: lrn1986 at gmail.com (LRN)
Date: Tue, 20 Jul 2010 09:23:03 +0400
Subject: Mingw compatibility patch
In-Reply-To: <20100719170912.6767D56425@rebar.astron.com>
References: <20100719170912.6767D56425@rebar.astron.com>
Message-ID: <4C4532B7.8050408@gmail.com>

  On 19.07.2010 21:09, Christos Zoulas wrote:
> On Jul 18,  6:32pm, lrn1986 at gmail.com (LRN) wrote:
> | Content-Transfer-Encoding: 7bit
> |
> |   Directory diff against file 5.04 is attached.
> | It is not perfect by any measure, especially the bits in configure.ac
> | and Makefile.am, and i've chosen to cull away missing functionality
> | instead of porting it, such as symlink support (MSVCRT does not seem to
> | be symlink-aware even if Windows is) and archive unpacking with
> | fork()/exec() (can be rewritten with spawn(), but that's
>
> I'd rather keep this around as a diff since I would prefer mingw32 to
> fix their own issues (printf c99 standard formats etc), rather than
> polluting the code.
1) There are ways to present these fixes in a nicer way, such as the way 
glib does it (and i can do that)
2) It will probably take some years for MinGW maintainers to add 
C99-conformant printf implementation, so it's not a temporary issue you 
can wait upon (actually, you can, but i wouldn't want you to)
3) There are other fixes which may actually go into code in their 
present form, no need to dump everything together. Though i have a bad 
habit of presenting independent patches in a single lump, especially in 
this particular case, since i didn't have any VCS to work with, just a 
simple recursive diff (and that i can fix by creating a local git 
repository out of a file(1) source snapshot)
4) How exactly are you going to keep it in a diff form? Will people be 
able to learn about it and use it? Will it be applied automatically for 
a particular platform? Because the goal is to allow people to compile 
file(1) and libmagic out of the box, preferably without extra patching 
that needs to be done manually.
> | (with altered magic.mgc search paths, yay!).
> |
> | P.S. is there a way to prevent libtool from picking up compiled file.exe
> | from the source tree when calling 'make' again? Because if this file.exe
> | doesn't work (wich is often the case during development), libtool's
> | magic tests will fail, and libtool will complain about missing real
> | libraries.
>
> This is not desirable again because if the version of the magic file
> changes with the new development code, building the magic file with
> the old executable will not work.
How could the version of the magic file that is located within the 
source tree (and should be preferably compiled with the version of the 
file program built from the same source tree, as you have said) be 
connected with actual magic file installed in the system along with the 
file program actually used to do things? Either i don't know something 
about the shell/libtool/gcc (which is probable), or you misunderstood my 
post-scriptum (which is also probable).
I think, actually, that on *nix this problem can be (and probably is) 
dodged by chmod'ing newly created executables as non-executable (aren't 
they made that way by default?), preventing shell from picking them up 
when libtool runs a `file` command.


From mdorey at bluearc.com  Tue Jul 20 10:20:26 2010
From: mdorey at bluearc.com (Martin Dorey)
Date: Tue, 20 Jul 2010 00:20:26 -0700
Subject: Mingw compatibility patch
Message-ID: <54A098E33E92A04EB0DD9A2E8B546CB00478AB5485@us-ex-mbx1.terastack.bluearc.com>

> 2) It will probably take some years for 
> MinGW maintainers to add 
> C99-conformant printf implementation

I doubt they'll ever do it.  Their project is not about recreating a Linux-like environment on Windows (that's Cygwin).  Their project is about being able to compile with gcc for Windows, using Microsoft's C runtime.  Portability from Microsoft's compilers is quite possibly more important to them than portability from other platforms.  It's Microsoft's printf, then, that's causing the issue.  Perhaps Microsoft will eventually get with the C99 program but I wouldn't hold your breath there either.

----- Original Message -----
From: file-bounces at mx.gw.com <file-bounces at mx.gw.com>
To: file at mx.gw.com <file at mx.gw.com>
Sent: Mon Jul 19 22:23:03 2010
Subject: Re: Mingw compatibility patch

  On 19.07.2010 21:09, Christos Zoulas wrote:
> On Jul 18,  6:32pm, lrn1986 at gmail.com (LRN) wrote:
> | Content-Transfer-Encoding: 7bit
> |
> |   Directory diff against file 5.04 is attached.
> | It is not perfect by any measure, especially the bits in configure.ac
> | and Makefile.am, and i've chosen to cull away missing functionality
> | instead of porting it, such as symlink support (MSVCRT does not seem to
> | be symlink-aware even if Windows is) and archive unpacking with
> | fork()/exec() (can be rewritten with spawn(), but that's
>
> I'd rather keep this around as a diff since I would prefer mingw32 to
> fix their own issues (printf c99 standard formats etc), rather than
> polluting the code.
1) There are ways to present these fixes in a nicer way, such as the way 
glib does it (and i can do that)
2) It will probably take some years for MinGW maintainers to add 
C99-conformant printf implementation, so it's not a temporary issue you 
can wait upon (actually, you can, but i wouldn't want you to)
3) There are other fixes which may actually go into code in their 
present form, no need to dump everything together. Though i have a bad 
habit of presenting independent patches in a single lump, especially in 
this particular case, since i didn't have any VCS to work with, just a 
simple recursive diff (and that i can fix by creating a local git 
repository out of a file(1) source snapshot)
4) How exactly are you going to keep it in a diff form? Will people be 
able to learn about it and use it? Will it be applied automatically for 
a particular platform? Because the goal is to allow people to compile 
file(1) and libmagic out of the box, preferably without extra patching 
that needs to be done manually.
> | (with altered magic.mgc search paths, yay!).
> |
> | P.S. is there a way to prevent libtool from picking up compiled file.exe
> | from the source tree when calling 'make' again? Because if this file.exe
> | doesn't work (wich is often the case during development), libtool's
> | magic tests will fail, and libtool will complain about missing real
> | libraries.
>
> This is not desirable again because if the version of the magic file
> changes with the new development code, building the magic file with
> the old executable will not work.
How could the version of the magic file that is located within the 
source tree (and should be preferably compiled with the version of the 
file program built from the same source tree, as you have said) be 
connected with actual magic file installed in the system along with the 
file program actually used to do things? Either i don't know something 
about the shell/libtool/gcc (which is probable), or you misunderstood my 
post-scriptum (which is also probable).
I think, actually, that on *nix this problem can be (and probably is) 
dodged by chmod'ing newly created executables as non-executable (aren't 
they made that way by default?), preventing shell from picking them up 
when libtool runs a `file` command.

_______________________________________________
File mailing list
File at mx.gw.com
http://mx.gw.com/mailman/listinfo/file

From christos at zoulas.com  Tue Jul 20 20:51:11 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 20 Jul 2010 13:51:11 -0400
Subject: Mingw compatibility patch
In-Reply-To: <4C4532B7.8050408@gmail.com>
       from LRN (Jul 20,  9:23am)
Message-ID: <20100720175111.572C256425@rebar.astron.com>

On Jul 20,  9:23am, lrn1986 at gmail.com (LRN) wrote:
-- Subject: Re: Mingw compatibility patch

| 1) There are ways to present these fixes in a nicer way, such as the way 
| glib does it (and i can do that)

The fewer ifdefs in the code the better.

| 2) It will probably take some years for MinGW maintainers to add 
| C99-conformant printf implementation, so it's not a temporary issue you 
| can wait upon (actually, you can, but i wouldn't want you to)

Well, it is 2010 -- it is already 11 years since the standard was published,
what are they waiting on? By the time they finish, there will be another
c standard published.

| 3) There are other fixes which may actually go into code in their 
| present form, no need to dump everything together. Though i have a bad 
| habit of presenting independent patches in a single lump, especially in 
| this particular case, since i didn't have any VCS to work with, just a 
| simple recursive diff (and that i can fix by creating a local git 
| repository out of a file(1) source snapshot)

I don't mind the single lump.

| 4) How exactly are you going to keep it in a diff form? Will people be 
| able to learn about it and use it? Will it be applied automatically for 
| a particular platform? Because the goal is to allow people to compile 
| file(1) and libmagic out of the box, preferably without extra patching 
| that needs to be done manually.

I would just add the diff file to the distribution in README.MIGWIN

| How could the version of the magic file that is located within the 
| source tree (and should be preferably compiled with the version of the 
| file program built from the same source tree, as you have said) be 
| connected with actual magic file installed in the system along with the 
| file program actually used to do things? Either i don't know something 
| about the shell/libtool/gcc (which is probable), or you misunderstood my 
| post-scriptum (which is also probable).
| I think, actually, that on *nix this problem can be (and probably is) 
| dodged by chmod'ing newly created executables as non-executable (aren't 
| they made that way by default?), preventing shell from picking them up 
| when libtool runs a `file` command.

The binary you just compiled is used to compile the magic file in the source
tree. It needs to be the binary associated with the same version of the
file.

christos


From lrn1986 at gmail.com  Tue Jul 20 22:18:59 2010
From: lrn1986 at gmail.com (LRN)
Date: Tue, 20 Jul 2010 23:18:59 +0400
Subject: Mingw compatibility patch
In-Reply-To: <20100720175111.572C256425@rebar.astron.com>
References: <20100720175111.572C256425@rebar.astron.com>
Message-ID: <4C45F6A3.1040405@gmail.com>

  On 20.07.2010 21:51, Christos Zoulas wrote:
> On Jul 20,  9:23am, lrn1986 at gmail.com (LRN) wrote:
> -- Subject: Re: Mingw compatibility patch
>
> | 1) There are ways to present these fixes in a nicer way, such as the way
> | glib does it (and i can do that)
>
> The fewer ifdefs in the code the better.
How about a single:
#ifdef WIN32
#define SIZE_T_FORMAT ""
#else
#define SIZE_T_FORMAT "z"
#endif

And then instead of:
printf ("There are %zu thingies", some_sizeof);
do it this way:
printf ("There are %" SIZE_T_FORMAT "u thingies", some_sizeof);

That should pre-process into:
printf ("There are %zu thingies", some_sizeof);
or
printf ("There are %u thingies", some_sizeof);
Depending on the platform.

Glib has a special /usr/lib/glib-2.0/include/glibconfig.h header for 
such platform-specific code, but i can't imagine libmagic doing 
something like that, so that might just go into some header other than 
magic.h
> | 2) It will probably take some years for MinGW maintainers to add
> | C99-conformant printf implementation, so it's not a temporary issue you
> | can wait upon (actually, you can, but i wouldn't want you to)
>
> Well, it is 2010 -- it is already 11 years since the standard was published,
> what are they waiting on? By the time they finish, there will be another
> c standard published.
What is Microsoft waiting for? For Windows to dominate the market and 
wipe out POSIX competition. Which won't happen
What are MinGW devs waiting for? Hard to tell. Martin already pointed 
out that such a task might be outside their scope. And even if it isn't, 
they are VERY peculiar about code licensing (MinGW code must be 100% 
public domain, and must derive 100% from public documentation; although 
the latter is not a problem in this particular case)
>   
> | 4) How exactly are you going to keep it in a diff form? Will people be
> | able to learn about it and use it? Will it be applied automatically for
> | a particular platform? Because the goal is to allow people to compile
> | file(1) and libmagic out of the box, preferably without extra patching
> | that needs to be done manually.
>
> I would just add the diff file to the distribution in README.MIGWIN
Oh, well, let's hope people do read such things. I know i do not :)
> | How could the version of the magic file that is located within the
> | source tree (and should be preferably compiled with the version of the
> | file program built from the same source tree, as you have said) be
> | connected with actual magic file installed in the system along with the
> | file program actually used to do things? Either i don't know something
> | about the shell/libtool/gcc (which is probable), or you misunderstood my
> | post-scriptum (which is also probable).
> | I think, actually, that on *nix this problem can be (and probably is)
> | dodged by chmod'ing newly created executables as non-executable (aren't
> | they made that way by default?), preventing shell from picking them up
> | when libtool runs a `file` command.
>
> The binary you just compiled is used to compile the magic file in the source
> tree. It needs to be the binary associated with the same version of the
> file.
Isn't it possible to invoke the just-compiled binary file(1) directly, 
by relative/absolute path (not relying on shell to find the right 
version), instead of invoking it by name only? And shove it somewhere 
deep, so that shell won't find it by itself?


From christos at zoulas.com  Tue Jul 20 22:37:19 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 20 Jul 2010 15:37:19 -0400
Subject: Mingw compatibility patch
In-Reply-To: <4C45F6A3.1040405@gmail.com>
       from LRN (Jul 20, 11:18pm)
Message-ID: <20100720193719.9EE0E56425@rebar.astron.com>

On Jul 20, 11:18pm, lrn1986 at gmail.com (LRN) wrote:
-- Subject: Re: Mingw compatibility patch

| How about a single:
| #ifdef WIN32
| #define SIZE_T_FORMAT ""
| #else
| #define SIZE_T_FORMAT "z"
| #endif
| 
| And then instead of:
| printf ("There are %zu thingies", some_sizeof);
| do it this way:
| printf ("There are %" SIZE_T_FORMAT "u thingies", some_sizeof);
| 
| That should pre-process into:
| printf ("There are %zu thingies", some_sizeof);
| or
| printf ("There are %u thingies", some_sizeof);
| Depending on the platform.

I think that would work, but might fail where size_t is "unsigned long".
Isn't that the case for WIN64?

| Glib has a special /usr/lib/glib-2.0/include/glibconfig.h header for 
| such platform-specific code, but i can't imagine libmagic doing 
| something like that, so that might just go into some header other than 
| magic.h

It can go to a private header.

| > | 4) How exactly are you going to keep it in a diff form? Will people be
| > | able to learn about it and use it? Will it be applied automatically for
| > | a particular platform? Because the goal is to allow people to compile
| > | file(1) and libmagic out of the box, preferably without extra patching
| > | that needs to be done manually.
| >
| > I would just add the diff file to the distribution in README.MIGWIN
| Oh, well, let's hope people do read such things. I know i do not :)

:-)
`
| > | How could the version of the magic file that is located within the
| > | source tree (and should be preferably compiled with the version of the
| > | file program built from the same source tree, as you have said) be
| > | connected with actual magic file installed in the system along with the
| > | file program actually used to do things? Either i don't know something
| > | about the shell/libtool/gcc (which is probable), or you misunderstood my
| > | post-scriptum (which is also probable).
| > | I think, actually, that on *nix this problem can be (and probably is)
| > | dodged by chmod'ing newly created executables as non-executable (aren't
| > | they made that way by default?), preventing shell from picking them up
| > | when libtool runs a `file` command.
| >
| > The binary you just compiled is used to compile the magic file in the source
| > tree. It needs to be the binary associated with the same version of the
| > file.
| Isn't it possible to invoke the just-compiled binary file(1) directly, 
| by relative/absolute path (not relying on shell to find the right 
| version), instead of invoking it by name only? And shove it somewhere 
| deep, so that shell won't find it by itself?

I think it already does that. He was worried about making
the build fail if there is a bug in the new binary, if I understand correctly.

christos


From j at hug.gs  Tue Jul 20 23:10:34 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 20 Jul 2010 13:10:34 -0700
Subject: Mingw compatibility patch
In-Reply-To: <20100720193719.9EE0E56425@rebar.astron.com>
References: <4C45F6A3.1040405@gmail.com>
	<20100720193719.9EE0E56425@rebar.astron.com>
Message-ID: <AANLkTim0xjxUpaXH0uia2sExEP7ooZ4Czv_ncbt-5k9C@mail.gmail.com>

On Tue, Jul 20, 2010 at 12:37 PM, Christos Zoulas <christos at zoulas.com> wrote:
> | That should pre-process into:
> | printf ("There are %zu thingies", some_sizeof);
> | or
> | printf ("There are %u thingies", some_sizeof);
> | Depending on the platform.
>
> I think that would work, but might fail where size_t is "unsigned long".
> Isn't that the case for WIN64?

size_t is 8 bytes long on win64.  The equivalent of "z" is "I" or
"I64" for Microsoft's CRT.

  http://msdn.microsoft.com/en-us/library/tcxf1dw6(v=VS.71).aspx

For what it's worth, I think it might be easier to just build in a
small printf module from e.g. the Linux kernel or uclibc or vstr or
whatever.  Trying to make format strings portable is yet another
detour between memory and the screen, and most standards-compliant CRT
versions are huge enough as it is.


From christos at zoulas.com  Wed Jul 21 00:20:14 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 20 Jul 2010 17:20:14 -0400
Subject: Mingw compatibility patch
In-Reply-To: <AANLkTim0xjxUpaXH0uia2sExEP7ooZ4Czv_ncbt-5k9C@mail.gmail.com>
	from "Dr. Jesus" (Jul 20,  1:10pm)
Message-ID: <20100720212014.E806756425@rebar.astron.com>

On Jul 20,  1:10pm, j at hug.gs ("Dr. Jesus") wrote:
-- Subject: Re: Mingw compatibility patch

| On Tue, Jul 20, 2010 at 12:37 PM, Christos Zoulas <christos at zoulas.com> wrote:
| > | That should pre-process into:
| > | printf ("There are %zu thingies", some_sizeof);
| > | or
| > | printf ("There are %u thingies", some_sizeof);
| > | Depending on the platform.
| >
| > I think that would work, but might fail where size_t is "unsigned long".
| > Isn't that the case for WIN64?
| 
| size_t is 8 bytes long on win64.  The equivalent of "z" is "I" or
| "I64" for Microsoft's CRT.
| 
|   http://msdn.microsoft.com/en-us/library/tcxf1dw6(v=VS.71).aspx
| 
| For what it's worth, I think it might be easier to just build in a
| small printf module from e.g. the Linux kernel or uclibc or vstr or
| whatever.  Trying to make format strings portable is yet another
| detour between memory and the screen, and most standards-compliant CRT
| versions are huge enough as it is.

there is the ugly and portable solution to use "%lu", (unsigned long)size_t_var

christos


From j at hug.gs  Wed Jul 21 00:27:33 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 20 Jul 2010 14:27:33 -0700
Subject: Mingw compatibility patch
In-Reply-To: <20100720212014.E806756425@rebar.astron.com>
References: <AANLkTim0xjxUpaXH0uia2sExEP7ooZ4Czv_ncbt-5k9C@mail.gmail.com>
	<20100720212014.E806756425@rebar.astron.com>
Message-ID: <AANLkTinEAZ2eek1QvKu2dWffosnRhf8xBiR-_XSF7Eav@mail.gmail.com>

On Tue, Jul 20, 2010 at 2:20 PM, Christos Zoulas <christos at zoulas.com> wrote:
> On Jul 20, ?1:10pm, j at hug.gs ("Dr. Jesus") wrote:
> -- Subject: Re: Mingw compatibility patch
>
> | On Tue, Jul 20, 2010 at 12:37 PM, Christos Zoulas <christos at zoulas.com> wrote:
> | > | That should pre-process into:
> | > | printf ("There are %zu thingies", some_sizeof);
> | > | or
> | > | printf ("There are %u thingies", some_sizeof);
> | > | Depending on the platform.
> | >
> | > I think that would work, but might fail where size_t is "unsigned long".
> | > Isn't that the case for WIN64?
> |
> | size_t is 8 bytes long on win64. ?The equivalent of "z" is "I" or
> | "I64" for Microsoft's CRT.
> |
> | ? http://msdn.microsoft.com/en-us/library/tcxf1dw6(v=VS.71).aspx
> |
> | For what it's worth, I think it might be easier to just build in a
> | small printf module from e.g. the Linux kernel or uclibc or vstr or
> | whatever. ?Trying to make format strings portable is yet another
> | detour between memory and the screen, and most standards-compliant CRT
> | versions are huge enough as it is.
>
> there is the ugly and portable solution to use "%lu", (unsigned long)size_t_var

Sure, until you build on a platform where sizeof(size_t) !=
sizeof(unsigned long).  All new macs are in this boat:

$ file `which file`
/usr/bin/file: Mach-O universal binary with 2 architectures
/usr/bin/file (for architecture x86_64):        Mach-O 64-bit executable x86_64
/usr/bin/file (for architecture i386):  Mach-O executable i386
$ uname -a
Darwin hurricane-2.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr
23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 i386

One could use %llu instead, but then you need macros to abstract the
format string and the cast.


From lrn1986 at gmail.com  Wed Jul 21 00:28:14 2010
From: lrn1986 at gmail.com (LRN)
Date: Wed, 21 Jul 2010 01:28:14 +0400
Subject: Mingw compatibility patch
In-Reply-To: <20100720193719.9EE0E56425@rebar.astron.com>
References: <20100720193719.9EE0E56425@rebar.astron.com>
Message-ID: <4C4614EE.4070907@gmail.com>

  On 20.07.2010 23:37, Christos Zoulas wrote:
> On Jul 20, 11:18pm, lrn1986 at gmail.com (LRN) wrote:
> -- Subject: Re: Mingw compatibility patch
>
> | How about a single:
> | #ifdef WIN32
> | #define SIZE_T_FORMAT ""
> | #else
> | #define SIZE_T_FORMAT "z"
> | #endif
> |
> | And then instead of:
> | printf ("There are %zu thingies", some_sizeof);
> | do it this way:
> | printf ("There are %" SIZE_T_FORMAT "u thingies", some_sizeof);
> |
> | That should pre-process into:
> | printf ("There are %zu thingies", some_sizeof);
> | or
> | printf ("There are %u thingies", some_sizeof);
> | Depending on the platform.
>
> I think that would work, but might fail where size_t is "unsigned long".
It is. However, this simple testcase:

#include <stdio.h>
int main (int argc, char **argv) {
   size_t s = (size_t) -1;
   printf ("There are %lu thingies", s);
   return 0;
}

produces a warning ( warning: format '%lu' expects type 'long unsigned 
int', but argument 2 has type 'size_t' ) when compiled with -Wall, and 
does not produce one if i change "%lu" to "%u".
Both will produce the same output:
There are 4294967295 thingies
> Isn't that the case for WIN64?
OK, MinGW defines

#define __SIZE_TYPE__ long unsigned int

while MinGW64 defines it as

#ifdef _WIN64
#define __SIZE_TYPE__ long long unsigned int
#else
#define __SIZE_TYPE__ long unsigned int
#endif

The above testcase does, when compiled with MinGW64, not produce a 
warning only when "%I64u" format is used (can't verify that the output 
is correct, because there are some errors during the compilation that i 
can't fix and don't care to fix, because i'm not interested in using 
MinGW64 at the moment).

So unless there are other varieties, this can be done as:

#ifdef WIN32
   #ifdef _WIN64
     #define SIZE_T_FORMAT "I64"
   #else
     #define SIZE_T_FORMAT ""
   #endif
#else
   #define SIZE_T_FORMAT "z"
#endif


> | Glib has a special /usr/lib/glib-2.0/include/glibconfig.h header for
> | such platform-specific code, but i can't imagine libmagic doing
> | something like that, so that might just go into some header other than
> | magic.h
>
> It can go to a private header.
file.h should do, it is included in all source files that use "%z" format
> |>  | How could the version of the magic file that is located within the
> |>  | source tree (and should be preferably compiled with the version of the
> |>  | file program built from the same source tree, as you have said) be
> |>  | connected with actual magic file installed in the system along with the
> |>  | file program actually used to do things? Either i don't know something
> |>  | about the shell/libtool/gcc (which is probable), or you misunderstood my
> |>  | post-scriptum (which is also probable).
> |>  | I think, actually, that on *nix this problem can be (and probably is)
> |>  | dodged by chmod'ing newly created executables as non-executable (aren't
> |>  | they made that way by default?), preventing shell from picking them up
> |>  | when libtool runs a `file` command.
> |>
> |>  The binary you just compiled is used to compile the magic file in the source
> |>  tree. It needs to be the binary associated with the same version of the
> |>  file.
> | Isn't it possible to invoke the just-compiled binary file(1) directly,
> | by relative/absolute path (not relying on shell to find the right
> | version), instead of invoking it by name only? And shove it somewhere
> | deep, so that shell won't find it by itself?
>
> I think it already does that.
If it already does that, why do i get file(1) from the source tree 
called by libtool?
...Wait a second...How exactly does shell look for executables on *nix? 
I remember reading somewhere that the fact that MSys shell looks for 
executables first in '.' is a non-POSIX behaviour and is done to 
maintain compatibility with Win32.
*Checks on Debian*
Yes, that's it. I've created two myscript.sh - one in my current (home) 
directory, and another in /bin/ directory, each one will print its 
location upon execution. When i invoke 'myscript.sh', the shell reads 
/bin/myscript.sh first; i have to use './myscript.sh' to invoke myscript 
in current directory. I see why people insist on calling things in 
current directory with './' prefix, and why that never seemed to make a 
difference in Msys - because on MSys $PATH begins with 
'.:/usr/local/bin:/mingw/bin:/bin' by default.

I think this is fixable on file(1) side (use different executable name, 
rename it to 'file' on installation). It is also possible to fix MSys by 
putting '.' into the end of the PATH, but that might have unforeseen 
consequences ?, because '.' is the first entry in the PATH by default 
(if i am not mistaken), and people tend to rely on defaults...


From christos at zoulas.com  Wed Jul 21 01:00:27 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 20 Jul 2010 18:00:27 -0400
Subject: Mingw compatibility patch
In-Reply-To: <AANLkTinEAZ2eek1QvKu2dWffosnRhf8xBiR-_XSF7Eav@mail.gmail.com>
	from "Dr. Jesus" (Jul 20,  2:27pm)
Message-ID: <20100720220028.043995653E@rebar.astron.com>

On Jul 20,  2:27pm, j at hug.gs ("Dr. Jesus") wrote:
-- Subject: Re: Mingw compatibility patch

| > there is the ugly and portable solution to use "%lu", (unsigned long)size=
| _t_var
| 
| Sure, until you build on a platform where sizeof(size_t) !=3D
| sizeof(unsigned long).  All new macs are in this boat:
| 
| $ file `which file`
| /usr/bin/file: Mach-O universal binary with 2 architectures
| /usr/bin/file (for architecture x86_64):        Mach-O 64-bit executable x8=
| 6_64
| /usr/bin/file (for architecture i386):  Mach-O executable i386
| $ uname -a
| Darwin hurricane-2.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr
| 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 i386
| 
| One could use %llu instead, but then you need macros to abstract the
| format string and the cast.

Really MacOS is not LP64?

christos


From j at hug.gs  Wed Jul 21 01:32:06 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 20 Jul 2010 15:32:06 -0700
Subject: Mingw compatibility patch
In-Reply-To: <20100720220028.043995653E@rebar.astron.com>
References: <AANLkTinEAZ2eek1QvKu2dWffosnRhf8xBiR-_XSF7Eav@mail.gmail.com>
	<20100720220028.043995653E@rebar.astron.com>
Message-ID: <AANLkTinN_DiTMcF9yjseAMsN87NuiHtZIfHAvSS0NNnB@mail.gmail.com>

On Tue, Jul 20, 2010 at 3:00 PM, Christos Zoulas <christos at zoulas.com> wrote:
> On Jul 20, ?2:27pm, j at hug.gs ("Dr. Jesus") wrote:
> -- Subject: Re: Mingw compatibility patch
>
> | > there is the ugly and portable solution to use "%lu", (unsigned long)size=
> | _t_var
> |
> | Sure, until you build on a platform where sizeof(size_t) !=3D
> | sizeof(unsigned long). ?All new macs are in this boat:
> |
> | $ file `which file`
> | /usr/bin/file: Mach-O universal binary with 2 architectures
> | /usr/bin/file (for architecture x86_64): ? ? ? ?Mach-O 64-bit executable x8=
> | 6_64
> | /usr/bin/file (for architecture i386): ?Mach-O executable i386
> | $ uname -a
> | Darwin hurricane-2.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr
> | 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 i386
> |
> | One could use %llu instead, but then you need macros to abstract the
> | format string and the cast.
>
> Really MacOS is not LP64?

When you put it like that it sounds sort of crazy, so I double-checked:

$ cat test13.c
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
        printf("%lu %lu\n", sizeof(size_t), sizeof(unsigned long));
        return EXIT_SUCCESS;
}
$ ./test13
8 8
$ file test13
test13: Mach-O 64-bit executable x86_64

Whoops.  I must have been thinking of Windows, where DWORDs are
unsigned longs and have to be 32 bit or all kinds of things would
break.


From jkaluza at redhat.com  Wed Jul 21 12:30:35 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Wed, 21 Jul 2010 11:30:35 +0200
Subject: Binary pattern vs Text pattern ideas
Message-ID: <201007211130.35921.jkaluza@redhat.com>

Hi,

sorry if I'm reinventing the wheel by this email, but File is quite old 
project and I wasn't able to find if somebody sent the idea I want to discuss 
about.

Currently File distinguish between binary magic pattern and text magic 
pattern. This brings some problems, because binary patterns are tried as first 
and sometimes they are too general.

For example following pattern .... :
0	string		#!/usr/bin/env		a
>15	string		>\0			%s script text executable

... is matched before:
0	search/1	#!\/usr/bin/env\ python	Python script text executable

In that example it's hard to fix that by using "search" instead of "string", 
because we need to get the name of interpreter somehow. If I'm missing 
something here and It's doable, please correct me.

The same (and in my opinion worse) situation is with search/xxxx/b patterns in 
sgml. This leads to having some scripts to be identified as HTML documented 
just because sgml uses binary search which is preferred.

I think there are three solutions for this situation:

1) Convert all binary patterns, which are currently used to match text files, 
into text patterns and use binary patterns only if it's really needed for 
binary formats (I don't think search/b is needed for example in "sgml"). This 
could fix lot of bugs, but it would not fix the "string" pattern example I've 
pasted above.

2) Handle binary and text patterns together and give them priority according 
to "strength". I think this is better solution. It's basically the same as 
treating text patterns as binary patterns. The problem here could be a 
performance degradation, because we will have to check more patterns for 
binary files, but for text files it could be faster, because we don't have to go 
through all binary patterns. I think File would match better with this change

However, it will still not fix some too general patterns. Imagine for example 
that you have unified diff of html file. You can write very good pattern to 
recognize diff and because unified diff is very unique format, it will almost 
always match just for a diff. Problem is it has low strength to beat:

#0	search/4096/cwbt	\<!--			exported SGML document text

so you have to add !:strength + 30 to fix that.

It's really hard to check the strength for every pattern to be sure your 
pattern is in good place in the patterns list. Also, the computed strength 
doesn't say anything about "how well the pattern match", because File doesn't 
know the format.

3) Add !:level option. This is based on 2), but fixes !:strength problem. The 
idea is that apprentice_sort function would prefer patterns with higher level 
over the patterns with the lower one. If the default level is for example 100, 
we can set level of that very good diff pattern to 101 and level of those 
"general" sgml patterns to 99. Patterns on the same level would be still 
sorted by strength.

Basically with "level" option we can set higher priority for patterns which 
match almost 100% and lower for patterns which are more general. I think it 
would be better than just relying on computed "strength" attribute.

Sorry for the long post...

Thanks for response,
Jan Kaluza


From kimmo at suominen.com  Wed Jul 21 15:14:27 2010
From: kimmo at suominen.com (Kimmo Suominen)
Date: Wed, 21 Jul 2010 08:14:27 -0400
Subject: Mingw compatibility patch
In-Reply-To: <4C4614EE.4070907@gmail.com>
References: <20100720193719.9EE0E56425@rebar.astron.com>
	<4C4614EE.4070907@gmail.com>
Message-ID: <AANLkTinmfy4a5F_NqAdM6jmu5ESygnxJhIWX7d0vV_SA@mail.gmail.com>

On Tue, Jul 20, 2010 at 17:28, LRN <lrn1986 at gmail.com> wrote:

> I think this is fixable on file(1) side (use different executable name,
> rename it to 'file' on installation). It is also possible to fix MSys by
> putting '.' into the end of the PATH, but that might have unforeseen
> consequences ?, because '.' is the first entry in the PATH by default (if i
> am not mistaken), and people tend to rely on defaults...
>

Having '.' in the beginning of your path is a security issue: anything you
have in your working directory that clashes with a standard command usually
found elsewhere in PATH would get executed instead of the standard command.

I don't think it is feasible to fix the build environment of every software
package to account for such a misconfiguration of the environment.

+ Kim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100721/99d84fff/attachment.html>

From lrn1986 at gmail.com  Wed Jul 21 15:51:36 2010
From: lrn1986 at gmail.com (LRN)
Date: Wed, 21 Jul 2010 16:51:36 +0400
Subject: Mingw compatibility patch
In-Reply-To: <AANLkTinmfy4a5F_NqAdM6jmu5ESygnxJhIWX7d0vV_SA@mail.gmail.com>
References: <20100720193719.9EE0E56425@rebar.astron.com>
	<4C4614EE.4070907@gmail.com>
	<AANLkTinmfy4a5F_NqAdM6jmu5ESygnxJhIWX7d0vV_SA@mail.gmail.com>
Message-ID: <4C46ED58.9050208@gmail.com>

  On 21.07.2010 16:14, Kimmo Suominen wrote:
> On Tue, Jul 20, 2010 at 17:28, LRN<lrn1986 at gmail.com>  wrote:
>
>> I think this is fixable on file(1) side (use different executable name,
>> rename it to 'file' on installation). It is also possible to fix MSys by
>> putting '.' into the end of the PATH, but that might have unforeseen
>> consequences ?, because '.' is the first entry in the PATH by default (if i
>> am not mistaken), and people tend to rely on defaults...
>>
> Having '.' in the beginning of your path is a security issue: anything you
> have in your working directory that clashes with a standard command usually
> found elsewhere in PATH would get executed instead of the standard command.
>
> I don't think it is feasible to fix the build environment of every software
> package to account for such a misconfiguration of the environment.
>
> + Kim
<quote file="MSYSROOT%/etc/profile" source="MSys base system package">
# My decision to add a . to the PATH and as the first item in the path list
# is to mimick the Win32 method of finding executables.
#
# I filter the PATH value setting in order to get ready for self hosting the
# MSYS runtime and wanting different paths searched first for files.
if [ $MSYSTEM == MINGW32 ]; then
   export PATH=".:/usr/local/bin:/mingw/bin:/bin:$PATH"
else
   export PATH=".:/usr/local/bin:/bin:/mingw/bin:$PATH"
fi
</quote>

This is a security issue indeed. Thing is, this security issue is 
hard-coded ( 
http://msdn.microsoft.com/en-us/library/ee872121%28v=VS.85%29.aspx and 
http://msdn.microsoft.com/en-us/library/ms682425%28VS.85%29.aspx ) into 
Win32 subsystem, and the whole OS works with this security issue in 
mind. Also, this security issue is enabled in MSys by default, and has 
been like this for years. So i wouldn't call it a 'misconfiguration'.

And certainly not every software package needs to run itself prior to 
installation (which is when the problem becomes apparent).

And this particular problem can't be fixed by cross-compiling from 
'well-configured' environment (i.e. a GNU/Linux box), because that 
environment won't be able to run file.exe that it cross-compiled (unless 
you're using Wine...)


From lrn1986 at gmail.com  Wed Jul 21 16:25:11 2010
From: lrn1986 at gmail.com (LRN)
Date: Wed, 21 Jul 2010 17:25:11 +0400
Subject: Mingw compatibility patch 2
In-Reply-To: <20100720193719.9EE0E56425@rebar.astron.com>
References: <20100720193719.9EE0E56425@rebar.astron.com>
Message-ID: <4C46F537.9080707@gmail.com>

  Changes from previous patch:
1) INT64_T_FORMAT and SIZE_T_FORMAT macros are used instead of %z and 
%ll. The macros themselves are defined in file.h (private header)
2) Fixed mingw-incompatible AC_STRUCT_TIMEZONE_DAYLIGHT in acinclude.m4 
(took some parts from upstream autoconf AC_STRUCT_TIMEZONE, as per 
http://osdir.com/ml/sysutils.autoconf.patches/2004-03/msg00049.html ), 
that eliminated two nasty warnings about tzname and daylight
3) Fixed PATHSEP to be ';' on Windows instead of ':' (otherwise 
driveletter in "X:\foo\bar" gets separated as a magic file name)
4) Changed some #ifdefs from __MINGW32__ to WIN32, since the code they 
guard is specific to the OS rather than toolkit (although i don't think 
anyone is ever going to compile file(1) with non-MinGW Windows compilers...)
5) Added libgnurx library check to configure.ac (MinGW only)
6) Added silent rules macro to configure.ac; Only works when certain 
files (which ones? I'm not sure) are re-generated from scratch, 
otherwise autotools 1.10 gets picked up (silent rules were introduced in 
1.11) and silent rules macro gets ignored (backward-compatible behaviour)
7) Included winsock2 header and #def'ed timespec as timeval (which is 
equivalent), instead of including pthread.h (which declares timespec by 
itself, properly)
8) Reduced number of #ifdefs (namely - HAVE_FORK) by lumping some pieces 
of code together

Possible improvements: Use __attribute__ ((unused)) more often instead 
of culling out variables and functions with #ifdef
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.diff
Type: text/x-patch
Size: 24524 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100721/d9e8bfd9/attachment-0001.bin>

From christos at zoulas.com  Wed Jul 21 18:12:07 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 21 Jul 2010 11:12:07 -0400
Subject: Mingw compatibility patch
In-Reply-To: <AANLkTinN_DiTMcF9yjseAMsN87NuiHtZIfHAvSS0NNnB@mail.gmail.com>
	from "Dr. Jesus" (Jul 20,  3:32pm)
Message-ID: <20100721151207.2880656426@rebar.astron.com>

On Jul 20,  3:32pm, j at hug.gs ("Dr. Jesus") wrote:
-- Subject: Re: Mingw compatibility patch

| When you put it like that it sounds sort of crazy, so I double-checked:
| 
| $ cat test13.c
| #include <stdio.h>
| #include <stdlib.h>
| 
| int main(void)
| {
|         printf("%lu %lu\n", sizeof(size_t), sizeof(unsigned long));
|         return EXIT_SUCCESS;
| }
| $ ./test13
| 8 8
| $ file test13
| test13: Mach-O 64-bit executable x86_64
| 
| Whoops.  I must have been thinking of Windows, where DWORDs are
| unsigned longs and have to be 32 bit or all kinds of things would
| break.

Yup, Windows is the only OS I know that is LLP64...

christos


From christos at zoulas.com  Wed Jul 21 19:48:17 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 21 Jul 2010 12:48:17 -0400
Subject: Mingw compatibility patch 2
In-Reply-To: <4C46F537.9080707@gmail.com>
       from LRN (Jul 21,  5:25pm)
Message-ID: <20100721164817.D8DFE56427@rebar.astron.com>

On Jul 21,  5:25pm, lrn1986 at gmail.com (LRN) wrote:
-- Subject: Re: Mingw compatibility patch 2

|   Changes from previous patch:
| 1) INT64_T_FORMAT and SIZE_T_FORMAT macros are used instead of %z and 
| %ll. The macros themselves are defined in file.h (private header)
| 2) Fixed mingw-incompatible AC_STRUCT_TIMEZONE_DAYLIGHT in acinclude.m4 
| (took some parts from upstream autoconf AC_STRUCT_TIMEZONE, as per 
| http://osdir.com/ml/sysutils.autoconf.patches/2004-03/msg00049.html ), 
| that eliminated two nasty warnings about tzname and daylight
| 3) Fixed PATHSEP to be ';' on Windows instead of ':' (otherwise 
| driveletter in "X:\foo\bar" gets separated as a magic file name)
| 4) Changed some #ifdefs from __MINGW32__ to WIN32, since the code they 
| guard is specific to the OS rather than toolkit (although i don't think 
| anyone is ever going to compile file(1) with non-MinGW Windows compilers...)
| 5) Added libgnurx library check to configure.ac (MinGW only)
| 6) Added silent rules macro to configure.ac; Only works when certain 
| files (which ones? I'm not sure) are re-generated from scratch, 
| otherwise autotools 1.10 gets picked up (silent rules were introduced in 
| 1.11) and silent rules macro gets ignored (backward-compatible behaviour)
| 7) Included winsock2 header and #def'ed timespec as timeval (which is 
| equivalent), instead of including pthread.h (which declares timespec by 
| itself, properly)
| 8) Reduced number of #ifdefs (namely - HAVE_FORK) by lumping some pieces 
| of code together
| 
| Possible improvements: Use __attribute__ ((unused)) more often instead 
| of culling out variables and functions with #ifdef

Thanks! I've applied it.

christos


From christos at zoulas.com  Wed Jul 21 20:09:03 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 21 Jul 2010 13:09:03 -0400
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201007211130.35921.jkaluza@redhat.com>
	from Jan Kaluza (Jul 21, 11:30am)
Message-ID: <20100721170903.64F1A56426@rebar.astron.com>

On Jul 21, 11:30am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Binary pattern vs Text pattern ideas

| Hi,
| 
| sorry if I'm reinventing the wheel by this email, but File is quite old 
| project and I wasn't able to find if somebody sent the idea I want to discuss 
| about.

No you are not. This is an active area of research :-) The separation of
text and binary matches is very recent, so is the strength and the search
pattern.

| Currently File distinguish between binary magic pattern and text magic 
| pattern. This brings some problems, because binary patterns are tried as first 
| and sometimes they are too general.
| 
| For example following pattern .... :
| 0	string		#!/usr/bin/env		a
| >15	string		>\0			%s script text executable
| 
| ... is matched before:
| 0	search/1	#!\/usr/bin/env\ python	Python script text executable
| 
| In that example it's hard to fix that by using "search" instead of "string", 
| because we need to get the name of interpreter somehow. If I'm missing 
| something here and It's doable, please correct me.
| 
| The same (and in my opinion worse) situation is with search/xxxx/b patterns in 
| sgml. This leads to having some scripts to be identified as HTML documented 
| just because sgml uses binary search which is preferred.
| 
| I think there are three solutions for this situation:
| 
| 1) Convert all binary patterns, which are currently used to match text files, 
| into text patterns and use binary patterns only if it's really needed for 
| binary formats (I don't think search/b is needed for example in "sgml"). This 
| could fix lot of bugs, but it would not fix the "string" pattern example I've 
| pasted above.

This is difficult because "text" files are hard to detect, unless you mean
ascii text of course. In particular web pages do not necessary contain only
ascii text.

| 2) Handle binary and text patterns together and give them priority according 
| to "strength". I think this is better solution. It's basically the same as 
| treating text patterns as binary patterns. The problem here could be a 
| performance degradation, because we will have to check more patterns for 
| binary files, but for text files it could be faster, because we don't have to go 
| through all binary patterns. I think File would match better with this change
| 
| However, it will still not fix some too general patterns. Imagine for example 
| that you have unified diff of html file. You can write very good pattern to 
| recognize diff and because unified diff is very unique format, it will almost 
| always match just for a diff. Problem is it has low strength to beat:
| 
| #0	search/4096/cwbt	\<!--			exported SGML document text
| 
| so you have to add !:strength + 30 to fix that.
| 
| It's really hard to check the strength for every pattern to be sure your 
| pattern is in good place in the patterns list. Also, the computed strength 
| doesn't say anything about "how well the pattern match", because File doesn't 
| know the format.

Well, another way is to say stronger than pattern "Y" to enforce a relative
matching.

| 3) Add !:level option. This is based on 2), but fixes !:strength problem. The 
| idea is that apprentice_sort function would prefer patterns with higher level 
| over the patterns with the lower one. If the default level is for example 100, 
| we can set level of that very good diff pattern to 101 and level of those 
| "general" sgml patterns to 99. Patterns on the same level would be still 
| sorted by strength.
| 
| Basically with "level" option we can set higher priority for patterns which 
| match almost 100% and lower for patterns which are more general. I think it 
| would be better than just relying on computed "strength" attribute.

I think that adding another knob will make things even more
complicated.  Before we commit to this, let's think carefully about
solutions to pattern match ordering.  Perhaps what we need is a
pretty-printer that will show what the strength of each pattern
is. The problem with doing that is that when/if we change the scoring
algorithm all the magic strength values will need to be re-visited.

christos

christos


From jkaluza at redhat.com  Thu Jul 22 10:36:43 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Thu, 22 Jul 2010 09:36:43 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <20100721170903.64F1A56426@rebar.astron.com>
References: <20100721170903.64F1A56426@rebar.astron.com>
Message-ID: <201007220936.43539.jkaluza@redhat.com>

On Wednesday, July 21, 2010 07:09:03 pm Christos Zoulas wrote:
> On Jul 21, 11:30am, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Binary pattern vs Text pattern ideas
> 
> | Hi,
> | 
> | sorry if I'm reinventing the wheel by this email, but File is quite old
> | project and I wasn't able to find if somebody sent the idea I want to
> | discuss about.
> 
> No you are not. This is an active area of research :-) The separation of
> text and binary matches is very recent, so is the strength and the search
> pattern.
> 
> | Currently File distinguish between binary magic pattern and text magic
> | pattern. This brings some problems, because binary patterns are tried as
> | first and sometimes they are too general.
> | 
> | For example following pattern .... :
> | 0	string		#!/usr/bin/env		a
> | 
> | >15	string		>\0			%s script text executable
> | 
> | ... is matched before:
> | 0	search/1	#!\/usr/bin/env\ python	Python script text executable
> | 
> | In that example it's hard to fix that by using "search" instead of
> | "string", because we need to get the name of interpreter somehow. If I'm
> | missing something here and It's doable, please correct me.
> | 
> | The same (and in my opinion worse) situation is with search/xxxx/b
> | patterns in sgml. This leads to having some scripts to be identified as
> | HTML documented just because sgml uses binary search which is preferred.
> | 
> | I think there are three solutions for this situation:
> | 
> | 1) Convert all binary patterns, which are currently used to match text
> | files, into text patterns and use binary patterns only if it's really
> | needed for binary formats (I don't think search/b is needed for example
> | in "sgml"). This could fix lot of bugs, but it would not fix the
> | "string" pattern example I've pasted above.
> 
> This is difficult because "text" files are hard to detect, unless you mean
> ascii text of course. In particular web pages do not necessary contain only
> ascii text.

My idea was more about writing "text" patterns only for "text" formats and 
binary patterns only for binary formats. So if you want to match html document 
(which definitely isn't binary format), there's no need to use binary pattern 
("search/xxxx/b). But that's not 100% doable, because sometimes you have to 
treat text format as binary (see the "#!/usr/bin/env" I've sent in previous 
mail).

So to sum it up, current basic problem is that "strength" doesn't work well, 
because binary patterns are always stronger than text patterns and there's no 
way to say "this text pattern should be tried before this binary pattern, 
because it's more precise".

In my opinion File should really handle both groups together (especially if 
there's no reason to not do it).

> | 2) Handle binary and text patterns together and give them priority
> | according to "strength". I think this is better solution. It's basically
> | the same as treating text patterns as binary patterns. The problem here
> | could be a performance degradation, because we will have to check more
> | patterns for binary files, but for text files it could be faster,
> | because we don't have to go through all binary patterns. I think File
> | would match better with this change
> | 
> | However, it will still not fix some too general patterns. Imagine for
> | example that you have unified diff of html file. You can write very good
> | pattern to recognize diff and because unified diff is very unique
> | format, it will almost always match just for a diff. Problem is it has
> | low strength to beat:
> | 
> | #0	search/4096/cwbt	\<!--			exported SGML document text
> | 
> | so you have to add !:strength + 30 to fix that.
> | 
> | It's really hard to check the strength for every pattern to be sure your
> | pattern is in good place in the patterns list. Also, the computed
> | strength doesn't say anything about "how well the pattern match",
> | because File doesn't know the format.
> 
> Well, another way is to say stronger than pattern "Y" to enforce a relative
> matching.

I think this would bring more difficulties, because sometimes/usually you want 
to define more patterns which should be stronger than Y and basically we would 
have to start identifying patterns uniquely to achieve this.

> | 3) Add !:level option. This is based on 2), but fixes !:strength problem.
> | The idea is that apprentice_sort function would prefer patterns with
> | higher level over the patterns with the lower one. If the default level
> | is for example 100, we can set level of that very good diff pattern to
> | 101 and level of those "general" sgml patterns to 99. Patterns on the
> | same level would be still sorted by strength.
> | 
> | Basically with "level" option we can set higher priority for patterns
> | which match almost 100% and lower for patterns which are more general. I
> | think it would be better than just relying on computed "strength"
> | attribute.
> 
> I think that adding another knob will make things even more
> complicated.  Before we commit to this, let's think carefully about
> solutions to pattern match ordering.

I've just come up with !:level, because that's why !:strength is there. if you 
use !:strength you want to say that this pattern is stronger than other ones, 
but "strength" is also computed by File automatically. So in my opinion 
"strength" has two meanings. The first one is to sort patterns automatically, 
the second one is to change the order if automatic sorting is not good. If you 
use !:strength, you always end with groups of patterns which are above another 
groups, but you have to check strength of every pattern you want to beat.

I'm not strictly against using !:strength for manual sorting and I really 
don't have to have !:level there, but I think it would make things clearer and 
it's closer to real use-case of !:strength option.

> Perhaps what we need is a
> pretty-printer that will show what the strength of each pattern
> is. The problem with doing that is that when/if we change the scoring
> algorithm all the magic strength values will need to be re-visited.

I think some option to show patterns list together with strength is really 
good idea. I will try to implement it and send you a patch.

> christos
> 
> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Jan Kaluza


From jkaluza at redhat.com  Fri Jul 23 11:22:27 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Fri, 23 Jul 2010 10:22:27 +0200
Subject: [PATCH] New "-l" option to show sorted patterns list
Message-ID: <201007231022.27107.jkaluza@redhat.com>

Hi

attached patch adds new "-l" option to show sorted patterns list together with 
each pattern's strength. It's written against File 5.04, but it uses 
SIZE_T_FORMAT from accepted patch in MinGW compatibility thread.

The goal of -l option is to let user/developer to check the "strength" 
attribute and the position of particular pattern in the patterns list easily.

Regards,
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-list-patterns.patch
Type: text/x-patch
Size: 5054 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100723/60ae8462/attachment.bin>

From lrn1986 at gmail.com  Sat Aug  7 10:03:59 2010
From: lrn1986 at gmail.com (LRN)
Date: Sat, 07 Aug 2010 11:03:59 +0400
Subject: Mingw compatibility patch 3
In-Reply-To: <20100721164817.D8DFE56427@rebar.astron.com>
References: <20100721164817.D8DFE56427@rebar.astron.com>
Message-ID: <4C5D055F.6020507@gmail.com>

  On 21.07.2010 20:48, Christos Zoulas wrote:
> On Jul 21,  5:25pm, lrn1986 at gmail.com (LRN) wrote:
> -- Subject: Re: Mingw compatibility patch 2
>
> |   Changes from previous patch:
> | 1) INT64_T_FORMAT and SIZE_T_FORMAT macros are used instead of %z and
> | %ll. The macros themselves are defined in file.h (private header)
> | 2) Fixed mingw-incompatible AC_STRUCT_TIMEZONE_DAYLIGHT in acinclude.m4
> | (took some parts from upstream autoconf AC_STRUCT_TIMEZONE, as per
> | http://osdir.com/ml/sysutils.autoconf.patches/2004-03/msg00049.html ),
> | that eliminated two nasty warnings about tzname and daylight
> | 3) Fixed PATHSEP to be ';' on Windows instead of ':' (otherwise
> | driveletter in "X:\foo\bar" gets separated as a magic file name)
> | 4) Changed some #ifdefs from __MINGW32__ to WIN32, since the code they
> | guard is specific to the OS rather than toolkit (although i don't think
> | anyone is ever going to compile file(1) with non-MinGW Windows compilers...)
> | 5) Added libgnurx library check to configure.ac (MinGW only)
> | 6) Added silent rules macro to configure.ac; Only works when certain
> | files (which ones? I'm not sure) are re-generated from scratch,
> | otherwise autotools 1.10 gets picked up (silent rules were introduced in
> | 1.11) and silent rules macro gets ignored (backward-compatible behaviour)
> | 7) Included winsock2 header and #def'ed timespec as timeval (which is
> | equivalent), instead of including pthread.h (which declares timespec by
> | itself, properly)
> | 8) Reduced number of #ifdefs (namely - HAVE_FORK) by lumping some pieces
> | of code together
> |
> | Possible improvements: Use __attribute__ ((unused)) more often instead
> | of culling out variables and functions with #ifdef
>
> Thanks! I've applied it.
>
> christos
>
After tinkering with a bit i've found some bugs in my patch:
1) PathRemoveFileSpec() is used instead of PathRemoveFileSpecA(). Since 
the string is definitely not wchar_t*, A-variant should be enforced.
2) At first i didn't grok the way libmagic does get_default_magic(). 
Apparently, ".magic" is a directory name, which should contain magic.mgc 
(and i thought that .magic is the file with rules!).
3) At some point i simply mistyped hmagicpath instead of tmppath, and 
access() call always fails because of that.

Attached are diff against 5.04, as well as a diff against previous diff, 
so that you can see the changes (you should really set up a repository 
somewhere, working with diff manually is not fun).

P.S. Also, i have some genuine DOS executable files (also attached, in 
archive), and file(1) seems to be unable to guess their type correctly 
(detects it as GLF_BINARY_LSB_FIRST).

P.P.S. I've found a way to work around the %z format problem - defining 
_GNU_SOURCE enables non-ANSI (?) extensions, implemented by MinGW 
libraries (linked statically) in addition to the stock MSVCRT. Among 
them are custom printf() which supports %z.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sam.tar.bz2
Type: application/octet-stream
Size: 134661 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100807/5a4157b9/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file.diff_to_file2.diff
Type: text/x-patch
Size: 1683 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100807/5a4157b9/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file2.diff
Type: text/x-patch
Size: 24561 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100807/5a4157b9/attachment-0003.bin>

From mdorey at bluearc.com  Mon Aug  9 19:35:45 2010
From: mdorey at bluearc.com (Martin Dorey)
Date: Mon, 9 Aug 2010 09:35:45 -0700
Subject: Mingw compatibility patch 3
In-Reply-To: <4C5D055F.6020507@gmail.com>
References: <20100721164817.D8DFE56427@rebar.astron.com>
	<4C5D055F.6020507@gmail.com>
Message-ID: <54A098E33E92A04EB0DD9A2E8B546CB00478F8FB13@us-ex-mbx1.terastack.bluearc.com>

> _GNU_SOURCE enables non-ANSI (?) extensions

...

> custom printf() which supports %z



It's good to know about that custom printf, thanks.  I don't think the intention there was to suggest that the z length modifier is non-ANSI but, just to clear up that issue once and for all, I quote directly from ISO/IEC 9899:1999(E) "Programming languages - C", section 7.19.6.1, paragraph 7:



z Specifies that a following d, i, o, u, x, or X conversion specifier applies to a

  size_t or the corresponding signed integer type argument; or that a

  following n conversion specifier applies to a pointer to a signed integer type

  corresponding to size_t argument.



-----Original Message-----
From: file-bounces at mx.gw.com [mailto:file-bounces at mx.gw.com] On Behalf Of LRN
Sent: Saturday, August 07, 2010 00:04
To: file at mx.gw.com
Subject: Mingw compatibility patch 3



  On 21.07.2010 20:48, Christos Zoulas wrote:

> On Jul 21,  5:25pm, lrn1986 at gmail.com (LRN) wrote:

> -- Subject: Re: Mingw compatibility patch 2

>

> |   Changes from previous patch:

> | 1) INT64_T_FORMAT and SIZE_T_FORMAT macros are used instead of %z and

> | %ll. The macros themselves are defined in file.h (private header)

> | 2) Fixed mingw-incompatible AC_STRUCT_TIMEZONE_DAYLIGHT in acinclude.m4

> | (took some parts from upstream autoconf AC_STRUCT_TIMEZONE, as per

> | http://osdir.com/ml/sysutils.autoconf.patches/2004-03/msg00049.html ),

> | that eliminated two nasty warnings about tzname and daylight

> | 3) Fixed PATHSEP to be ';' on Windows instead of ':' (otherwise

> | driveletter in "X:\foo\bar" gets separated as a magic file name)

> | 4) Changed some #ifdefs from __MINGW32__ to WIN32, since the code they

> | guard is specific to the OS rather than toolkit (although i don't think

> | anyone is ever going to compile file(1) with non-MinGW Windows compilers...)

> | 5) Added libgnurx library check to configure.ac (MinGW only)

> | 6) Added silent rules macro to configure.ac; Only works when certain

> | files (which ones? I'm not sure) are re-generated from scratch,

> | otherwise autotools 1.10 gets picked up (silent rules were introduced in

> | 1.11) and silent rules macro gets ignored (backward-compatible behaviour)

> | 7) Included winsock2 header and #def'ed timespec as timeval (which is

> | equivalent), instead of including pthread.h (which declares timespec by

> | itself, properly)

> | 8) Reduced number of #ifdefs (namely - HAVE_FORK) by lumping some pieces

> | of code together

> |

> | Possible improvements: Use __attribute__ ((unused)) more often instead

> | of culling out variables and functions with #ifdef

>

> Thanks! I've applied it.

>

> christos

>

After tinkering with a bit i've found some bugs in my patch:

1) PathRemoveFileSpec() is used instead of PathRemoveFileSpecA(). Since

the string is definitely not wchar_t*, A-variant should be enforced.

2) At first i didn't grok the way libmagic does get_default_magic().

Apparently, ".magic" is a directory name, which should contain magic.mgc

(and i thought that .magic is the file with rules!).

3) At some point i simply mistyped hmagicpath instead of tmppath, and

access() call always fails because of that.



Attached are diff against 5.04, as well as a diff against previous diff,

so that you can see the changes (you should really set up a repository

somewhere, working with diff manually is not fun).



P.S. Also, i have some genuine DOS executable files (also attached, in

archive), and file(1) seems to be unable to guess their type correctly

(detects it as GLF_BINARY_LSB_FIRST).



P.P.S. I've found a way to work around the %z format problem - defining

_GNU_SOURCE enables non-ANSI (?) extensions, implemented by MinGW

libraries (linked statically) in addition to the stock MSVCRT. Among

them are custom printf() which supports %z.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100809/e0c85d83/attachment.html>

From christos at zoulas.com  Tue Aug 10 13:00:27 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 10 Aug 2010 06:00:27 -0400
Subject: Mingw compatibility patch 3
In-Reply-To: <4C5D055F.6020507@gmail.com>
       from LRN (Aug  7, 11:03am)
Message-ID: <20100810100027.95B5B56426@rebar.astron.com>

On Aug 7, 11:03am, lrn1986 at gmail.com (LRN) wrote:
-- Subject: Mingw compatibility patch 3

| Attached are diff against 5.04, as well as a diff against previous diff, 
| so that you can see the changes (you should really set up a repository 
| somewhere, working with diff manually is not fun).

Applied.

| P.S. Also, i have some genuine DOS executable files (also attached, in 
| archive), and file(1) seems to be unable to guess their type correctly 
| (detects it as GLF_BINARY_LSB_FIRST).

Index: magic/Magdir/mips
===================================================================
RCS file: /p/file/cvsroot/file/magic/Magdir/mips,v
retrieving revision 1.5
diff -u -u -r1.5 mips
--- magic/Magdir/mips	19 Sep 2009 16:28:10 -0000	1.5
+++ magic/Magdir/mips	10 Aug 2010 10:00:07 -0000
@@ -171,8 +171,12 @@
 # GLF is OpenGL stream encoding
 0	string	glfHeadMagic();		GLF_TEXT
 4	belong	0x7d000000		GLF_BINARY_LSB_FIRST
+!:strength -30
 4	belong	0x0000007d		GLF_BINARY_MSB_FIRST
+!:strength -30
 # GLS is OpenGL stream encoding; GLS is the successor of GLF
 0	string	glsBeginGLS(		GLS_TEXT
 4	belong	0x10000000		GLS_BINARY_LSB_FIRST
+!:strength -30
 4	belong	0x00000010		GLS_BINARY_MSB_FIRST
+!:strength -30
| 
| P.P.S. I've found a way to work around the %z format problem - defining 
| _GNU_SOURCE enables non-ANSI (?) extensions, implemented by MinGW 
| libraries (linked statically) in addition to the stock MSVCRT. Among 
| them are custom printf() which supports %z.

It's ok the way we have it now, I will leave it as is.

christos


From jkaluza at redhat.com  Tue Aug 10 14:47:57 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 10 Aug 2010 13:47:57 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201007220936.43539.jkaluza@redhat.com>
References: <20100721170903.64F1A56426@rebar.astron.com>
	<201007220936.43539.jkaluza@redhat.com>
Message-ID: <201008101347.57281.jkaluza@redhat.com>

Hi,

is there anything new regarding the handling binary patterns and text patterns 
together? I thought something like attached patch (it's not the clean 
solution, but I'm attaching it just to show what I mean). Do you think it's 
good idea or are there some disadvantages I don't see?

Regards,
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-patterns.patch
Type: text/x-patch
Size: 1020 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100810/91239db9/attachment.bin>

From christos at zoulas.com  Tue Aug 10 14:58:46 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 10 Aug 2010 07:58:46 -0400
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201008101347.57281.jkaluza@redhat.com>
	from Jan Kaluza (Aug 10,  1:47pm)
Message-ID: <20100810115846.40E10564C4@rebar.astron.com>

On Aug 10,  1:47pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: Binary pattern vs Text pattern ideas

| Hi,
| 
| is there anything new regarding the handling binary patterns and text patterns 
| together? I thought something like attached patch (it's not the clean 
| solution, but I'm attaching it just to show what I mean). Do you think it's 
| good idea or are there some disadvantages I don't see?
| 
| Regards,
| Jan Kaluza

I don't see any major disadvantage; this is the way it had been before
the latest split between binary and text patterns, but it will change
the semantics. Perhaps Reuben Thomas has something more to say about
it.

christos


From jkaluza at redhat.com  Tue Aug 10 15:14:58 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 10 Aug 2010 14:14:58 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <20100810115846.40E10564C4@rebar.astron.com>
References: <20100810115846.40E10564C4@rebar.astron.com>
Message-ID: <201008101414.58236.jkaluza@redhat.com>

On Tuesday, August 10, 2010 01:58:46 pm Christos Zoulas wrote:
> On Aug 10,  1:47pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: Binary pattern vs Text pattern ideas
> 
> | Hi,
> | 
> | is there anything new regarding the handling binary patterns and text
> | patterns together? I thought something like attached patch (it's not the
> | clean solution, but I'm attaching it just to show what I mean). Do you
> | think it's good idea or are there some disadvantages I don't see?
> | 
> | Regards,
> | Jan Kaluza
> 
> I don't see any major disadvantage; this is the way it had been before
> the latest split between binary and text patterns, but it will change
> the semantics. Perhaps Reuben Thomas has something more to say about
> it.

Yes, it will change the semantics, but it will also bring some benefits (it  
will fix the problems described in my previous emails). I plan to compare 
file(1) results before that change and after change and correct the mistakes 
(I'm currently working on script which will make it easier). Also, it will be 
easier to maintain patterns with file-5.04-list-patterns.patch from another 
thread (when I change it to list binary and text patterns together in one 
list).

By the way, why you've done that split? If other developers are not against, I 
will try write some "final" patch which will merge binary and text patterns 
together.

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file


From christos at zoulas.com  Tue Aug 10 21:51:29 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 10 Aug 2010 14:51:29 -0400
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201008101414.58236.jkaluza@redhat.com>
	from Jan Kaluza (Aug 10,  2:14pm)
Message-ID: <20100810185129.720D7564C4@rebar.astron.com>

On Aug 10,  2:14pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: Binary pattern vs Text pattern ideas

| Yes, it will change the semantics, but it will also bring some benefits (it  
| will fix the problems described in my previous emails). I plan to compare 
| file(1) results before that change and after change and correct the mistakes 
| (I'm currently working on script which will make it easier). Also, it will be 
| easier to maintain patterns with file-5.04-list-patterns.patch from another 
| thread (when I change it to list binary and text patterns together in one 
| list).
| 
| By the way, why you've done that split? If other developers are not against, I 
| will try write some "final" patch which will merge binary and text patterns 
| together.

The theory is that if you know if a file is text or binary and you know which
patterns should be applied for text and binary files, you are going to get
more precise matches. Consider strings in a binary file for example. It can
also be a speed advantage.

christos


From jkaluza at redhat.com  Wed Aug 11 10:27:39 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Wed, 11 Aug 2010 09:27:39 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <20100810185129.720D7564C4@rebar.astron.com>
References: <20100810185129.720D7564C4@rebar.astron.com>
Message-ID: <201008110927.39799.jkaluza@redhat.com>

On Tuesday, August 10, 2010 08:51:29 pm Christos Zoulas wrote:
> On Aug 10,  2:14pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: Binary pattern vs Text pattern ideas
> 
> | Yes, it will change the semantics, but it will also bring some benefits
> | (it will fix the problems described in my previous emails). I plan to
> | compare file(1) results before that change and after change and correct
> | the mistakes (I'm currently working on script which will make it
> | easier). Also, it will be easier to maintain patterns with
> | file-5.04-list-patterns.patch from another thread (when I change it to
> | list binary and text patterns together in one list).
> | 
> | By the way, why you've done that split? If other developers are not
> | against, I will try write some "final" patch which will merge binary and
> | text patterns together.
> 
> The theory is that if you know if a file is text or binary and you know
> which patterns should be applied for text and binary files, you are going
> to get more precise matches. Consider strings in a binary file for
> example. It can also be a speed advantage.

Well, that presumes there will be *just* text patterns or *just* binary 
patterns for one file format. That's idea is not bad, but we have to find out 
what to do with patterns like this (from varied.scripts):

0	string		#!\ /			a
>3	string		>\0			%s script text executable

Because "string" is binary pattern, but it's used to match text scripts, so 
now every pattern which will try to match any script has to be binary too. 
That's the only problem I see with the current situation. Can't we just add 
string/s for these cases?

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Jan Kaluza


From christos at zoulas.com  Wed Aug 11 11:31:34 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 11 Aug 2010 04:31:34 -0400
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201008110927.39799.jkaluza@redhat.com>
	from Jan Kaluza (Aug 11,  9:27am)
Message-ID: <20100811083134.D493B564C4@rebar.astron.com>

On Aug 11,  9:27am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: Binary pattern vs Text pattern ideas

| > which patterns should be applied for text and binary files, you are going
| > to get more precise matches. Consider strings in a binary file for
| > example. It can also be a speed advantage.
| 
| Well, that presumes there will be *just* text patterns or *just* binary 
| patterns for one file format. That's idea is not bad, but we have to find out 
| what to do with patterns like this (from varied.scripts):
| 
| 0	string		#!\ /			a
| >3	string		>\0			%s script text executable
| 
| Because "string" is binary pattern, but it's used to match text scripts, so 
| now every pattern which will try to match any script has to be binary too. 
| That's the only problem I see with the current situation. Can't we just add 
| string/s for these cases?

I think we could. I would like to see what Reuben has to say first.

christos


From jkaluza at redhat.com  Thu Aug 12 13:04:45 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Thu, 12 Aug 2010 12:04:45 +0200
Subject: [PATCH] Python 2.7 support
Message-ID: <201008121204.45746.jkaluza@redhat.com>

Hi,

patch at https://bugzilla.redhat.com/attachment.cgi?id=438386 adds Python 2.7 
byte-compiled files support.

Regards,
Jan Kaluza


From christos at zoulas.com  Thu Aug 12 19:16:56 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 12 Aug 2010 12:16:56 -0400
Subject: [PATCH] Python 2.7 support
In-Reply-To: <201008121204.45746.jkaluza@redhat.com>
	from Jan Kaluza (Aug 12, 12:04pm)
Message-ID: <20100812161656.D8DC0564C4@rebar.astron.com>

On Aug 12, 12:04pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: [PATCH] Python 2.7 support

| Hi,
| 
| patch at https://bugzilla.redhat.com/attachment.cgi?id=438386 adds Python 2.7 
| byte-compiled files support.
| 
| Regards,
| Jan Kaluza

Thanks, applied.

christos


From christos at zoulas.com  Fri Aug 13 19:13:14 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Fri, 13 Aug 2010 12:13:14 -0400
Subject: magic entry for BINK video files
In-Reply-To: <EC0016A92CFB644CB18C39DECDCE78E80EF185@HE110881.EMEA1.CDS.T-INTERNAL.COM>
	from <Joerg-Cyril.Hoehle@t-systems.com> (Aug 13,  3:07pm)
Message-ID: <20100813161314.848BD564C4@rebar.astron.com>

On Aug 13,  3:07pm, Joerg-Cyril.Hoehle at t-systems.com (<Joerg-Cyril.Hoehle at t-systems.com>) wrote:
-- Subject: magic entry for BINK video files

| Hi,
| 
| please remove the very superficial "Civilization 4 video" entry that
| sadly takes precedence over my much more comprehensive Bink Video
| description accepted 2 years ago (2008-10-01).
| 
| It falsely reports every Bink video as Civilization 4 video. Quite to
| the contrary, Civ4's videos obviously are Bink videos, revision i.
| 
| In case you decided to standardize on the #Extension marker,
| please add one for the Bink Video. Here's a patch (against Ubuntu Lucid).
| 

Fixed, thanks.

christos


From Andrew.Jackson at bl.uk  Thu Aug 26 16:47:47 2010
From: Andrew.Jackson at bl.uk (Jackson, Andrew)
Date: Thu, 26 Aug 2010 14:47:47 +0100
Subject: Minor issue with plain text, Pascal and names.h
Message-ID: <74C97E7DF5A7784D997217FF75D121660AEAB8DB@w2k3-bspex1>

Hi,
 
I've been attempting to use file (5.04) to determine the file type for large volumes of content, including emails stored as plain text. We've hit a minor problem, in that it seems the logic in names.h means that if a plain text file happens to have the word 'record' or 'program' near the top, it gets classed as Pascal source code instead of plain text. This behaviour also percolates through to the PHP fileinfo command (http://stackoverflow.com/questions/2345899/fileinfo-and-mime-types-ive-never-heard-of/).
 
Would it be possible to make the Pascal signature a bit more specific? I'm happy to help, so just let me know if this is something someone in particular should deal with, or if I should just propose a patch.
 
Cheers,
Andy Jackson

---
Digital Preservation Architect
eIS Digital Preservation Team
The British Library
http://www.bl.uk/dp
---


 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20100826/8042e1fa/attachment.html>

From tledouxfr at gmail.com  Wed Sep 15 20:20:23 2010
From: tledouxfr at gmail.com (Thomas Ledoux)
Date: Wed, 15 Sep 2010 19:20:23 +0200
Subject: [PATCH] Entry for wave64 and rf64
Message-ID: <AANLkTikD1Bw-ekdUY2FA8KX0dUFrOU-AmYbwXK++E++U@mail.gmail.com>

Hello,

here is a patch to add identification of extension of the RIFF format
(namely Sony Wave64 and RF64) to handle 64 bit size streams.

Regards
  Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: files5.04-wav64.patch
Type: application/octet-stream
Size: 1540 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20100915/c9ae04f1/attachment.obj>

From jkaluza at redhat.com  Mon Sep 20 20:28:32 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Mon, 20 Sep 2010 19:28:32 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <20100811083134.D493B564C4@rebar.astron.com>
References: <20100811083134.D493B564C4@rebar.astron.com>
Message-ID: <201009201928.33028.jkaluza@redhat.com>

On Wednesday, August 11, 2010 10:31:34 am Christos Zoulas wrote:
> On Aug 11,  9:27am, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: Binary pattern vs Text pattern ideas
> 
> | > which patterns should be applied for text and binary files, you are
> | > going to get more precise matches. Consider strings in a binary file
> | > for example. It can also be a speed advantage.
> | 
> | Well, that presumes there will be *just* text patterns or *just* binary
> | patterns for one file format. That's idea is not bad, but we have to find
> | out what to do with patterns like this (from varied.scripts):
> | 
> | 0	string		#!\ /			a
> | 
> | >3	string		>\0			%s script text executable
> | 
> | Because "string" is binary pattern, but it's used to match text scripts,
> | so now every pattern which will try to match any script has to be binary
> | too. That's the only problem I see with the current situation. Can't we
> | just add string/s for these cases?
> 
> I think we could. I would like to see what Reuben has to say first.

Hi,

is there anything new about it? Can't we do that without Reuben?

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Regards,
Jan Kaluza


From christos at zoulas.com  Mon Sep 20 22:10:51 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 20 Sep 2010 15:10:51 -0400
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201009201928.33028.jkaluza@redhat.com>
	from Jan Kaluza (Sep 20,  7:28pm)
Message-ID: <20100920191051.CF18456427@rebar.astron.com>

On Sep 20,  7:28pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: Binary pattern vs Text pattern ideas

| Hi,
| 
| is there anything new about it? Can't we do that without Reuben?

Reuben just sent mail this morning so let's wait a day or so.

christos


From lfile-list at thax.hardliners.org  Sun Sep 26 18:31:28 2010
From: lfile-list at thax.hardliners.org (Tobias Hoffmann)
Date: Sun, 26 Sep 2010 17:31:28 +0200
Subject: HTML everywhere
Message-ID: <4C9F6750.7030704@thax.hardliners.org>

Hi,

I've seen a similiar topic in the archive, but this is pretty annoying, 
as my less-pager uses file(1) determine the file type, which leads to 
crippled file, because they are /clearly/ not html, but file indentifies 
them as "HTML document text".

The relevant section seems to be (abbreviated)
0      search/4096/cWbt        \<!doctype\ html
0      search/4096/cwbt        \<head
0      search/4096/cwbt        \<title
0      search/4096/cwbt        \<html
0      search/4096/cwbt        \<script
0      search/4096/cwbt        \<style
0      search/4096/cwbt        \<table
0      search/4096/cwbt        \<a\ href=

It was mentioned, that these reduced the false-negative rate from 40% to 
1%  (60% to 99% success). But the false-positive rate is equally 
important -- otherwise even
--- /usr/bin/file ---
echo $1 "HTML document text"
---
would be an improvement ...

Ok, lets look at some examples:
$ file ~/public_html/*.php     gives often HTML, instead of the more 
sensible PHP  -- AFAICS fixable by getting the priorities right
$ file  some_header.hpp       contains  std::vector<title_t> titles;    
-- fixable by using "</title>" as pattern, as for title the closing tag 
is not optional. I would suggest the same for "</script>", "</style>", 
"</table>". Even for "<head" I would suggest only allowing "<head>" or 
"<head ", the same for "<html>".
According to the standard "<title></title>" is REQUIRED for any 
html/xhtml document. So without "<title>" a document does, strictly 
speaking, not deserve to be called HTML!
But I do know that there are quite some HTML-files out there, that are 
everything a html needs to be, EXCEPT for a title.
Therefore I do agree that we need some heuristics. But they should be 
rather weak (low priority) indicators for html.

$ file /usr/include/c++/4.4/bits/*.h   -> seven files are incorrectly 
labeled "HTML", presumably because they contain "<a href=" as part of 
some (doxygen?) comments. Not good.
$ file some.xml     -> valid xml that contains <title>, but 
unfortunately starts with an unicode BOM -> not recognized as XML

$ file /usr/include/*/* | grep HTML     ... not funny.   
  e.g.   "QList<StyleName>" ...

I do appreciate all your work, and hope for further improvements.
Regards
  Tobias


From jkaluza at redhat.com  Wed Oct  6 15:58:43 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Wed, 6 Oct 2010 14:58:43 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <20100920191051.CF18456427@rebar.astron.com>
References: <20100920191051.CF18456427@rebar.astron.com>
Message-ID: <201010061458.43810.jkaluza@redhat.com>

On Monday, September 20, 2010 09:10:51 pm Christos Zoulas wrote:
> On Sep 20,  7:28pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: Binary pattern vs Text pattern ideas
> 
> | Hi,
> | 
> | is there anything new about it? Can't we do that without Reuben?
> 
> Reuben just sent mail this morning so let's wait a day or so.

Hi again,

attached patch adds string/t option, so it could be possible in the future to 
separate patterns for binary formats and text formats (so html will not need 
to use search/b and examples from this thread can use "string/t"). This fixes 
all problems described in this thread.

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Thanks for response,
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-text-string.patch
Type: text/x-patch
Size: 564 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101006/69e095a1/attachment.bin>

From christos at zoulas.com  Wed Oct  6 17:09:19 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 6 Oct 2010 10:09:19 -0400
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <201010061458.43810.jkaluza@redhat.com>
	from Jan Kaluza (Oct  6,  2:58pm)
Message-ID: <20101006140919.C9DDD564C4@rebar.astron.com>

On Oct 6,  2:58pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: Binary pattern vs Text pattern ideas

| index 58826c3..76f1748 100644
| --- a/src/apprentice.c
| +++ b/src/apprentice.c
| @@ -587,8 +587,12 @@ set_test_type(struct magic *mstart, struct magic *m)
|  	case FILE_PSTRING:
|  	case FILE_BESTRING16:
|  	case FILE_LESTRING16:
| -		/* binary test, set flag */
| -		mstart->flag |= BINTEST;
| +		if (mstart->str_flags & STRING_BINTEST)
| +			mstart->flag |= BINTEST;
| +		else if (mstart->str_flags & STRING_TEXTTEST)
| +			mstart->flag |= TEXTTEST;
| +		else
| +			mstart->flag |= BINTEST;
|  		break;
|  	case FILE_REGEX:
|  	case FILE_SEARCH:

We should just say then:

		/* Allow text overrides */
		if (mstart->str_flags & STRING_TEXTTEST)
			mstart->flag |= TEXTTEST;
		else
			mstart->flag |= BINTEST;

christos


From jkaluza at redhat.com  Wed Oct  6 18:27:34 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Wed, 6 Oct 2010 17:27:34 +0200
Subject: Binary pattern vs Text pattern ideas
In-Reply-To: <20101006140919.C9DDD564C4@rebar.astron.com>
References: <20101006140919.C9DDD564C4@rebar.astron.com>
Message-ID: <201010061727.34659.jkaluza@redhat.com>

On Wednesday, October 06, 2010 04:09:19 pm Christos Zoulas wrote:
> On Oct 6,  2:58pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: Binary pattern vs Text pattern ideas
> 
> | index 58826c3..76f1748 100644
> | --- a/src/apprentice.c
> | +++ b/src/apprentice.c
> | @@ -587,8 +587,12 @@ set_test_type(struct magic *mstart, struct magic *m)
> | 
> |  	case FILE_PSTRING:
> |  	case FILE_BESTRING16:
> | 
> |  	case FILE_LESTRING16:
> | -		/* binary test, set flag */
> | -		mstart->flag |= BINTEST;
> | +		if (mstart->str_flags & STRING_BINTEST)
> | +			mstart->flag |= BINTEST;
> | +		else if (mstart->str_flags & STRING_TEXTTEST)
> | +			mstart->flag |= TEXTTEST;
> | +		else
> | +			mstart->flag |= BINTEST;
> | 
> |  		break;
> |  	
> |  	case FILE_REGEX:
> 
> |  	case FILE_SEARCH:
> We should just say then:
> 
> 		/* Allow text overrides */
> 		if (mstart->str_flags & STRING_TEXTTEST)
> 			mstart->flag |= TEXTTEST;
> 		else
> 			mstart->flag |= BINTEST;

Ah, I see. You're right. Thanks for correction.

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Regards,
Jan Kaluza


From tanja.lewandovski at t-online.de  Sat Oct 16 17:27:17 2010
From: tanja.lewandovski at t-online.de (Roberto Maar)
Date: Sat, 16 Oct 2010 16:27:17 +0200
Subject: too many matches for text/x-fortran
Message-ID: <201010161627.17416.robi@uses.berlios.de>

Hello,

following problem:

In the magic file I found for fortran

# $File: fortran,v 1.6 2009/09/19 16:28:09 christos Exp $
# FORTRAN source
0       regex/100       \^[Cc][\ \t]    FORTRAN program
!:mime  text/x-fortran

I think this is aimed at the comments in the Fortran source code.
My program uses libmagic and magic_buffer() and analyzed millions of raw data
blocks per hour and has a lot of false hits for Fortran.

# file -v
file-5.04
magic file from /usr/local/share/misc/magic
# file -i *
0000003250.f: text/x-fortran; charset=binary
0000256550.f: text/x-fortran; charset=binary
0000259644.f: text/x-fortran; charset=binary
# hexdump -n32 -C 0000003250.f
00000000  c3 05 96 ea 79 05 50 f2  5e f7 f0 fb 28 de 7d 04  |....y.P.^...(.}.|
00000010  df c9 6d 09 be c7 72 01  1a c8 35 ed f1 b9 a4 e7  |..m...r...5.....|
# hexdump -n32 -C 0000256550.f
00000000  c3 05 e9 0c 85 05 dd 0c  12 05 50 0c 75 04 5a 0b  |..........P.u.Z.|
00000010  ac 03 24 0a b5 02 c0 08  72 01 53 07 e2 ff f5 05  |..$.....r.S.....|
# hexdump -n32 -C 0000259644.f
00000000  83 05 8e 08 97 05 ce 08  f6 11 98 16 8a 19 12 1e  |................|
00000010  bb 14 bd 17 72 0f b6 11  06 0f 78 11 bf 11 64 14  |....r.....x...d.|


on older file version (testet 4.17 and 5.03) the same are detected more 
accurately. 

# /usr/bin/file -v
file-5.03
magic file from /etc/magic:/usr/share/misc/magic
# /usr/bin/file -i *
0000003250.f: application/octet-stream; charset=binary
0000256550.f: application/octet-stream; charset=binary
0000259644.f: application/octet-stream; charset=binary

I can not understand the behavior. Why match a regular expression
"^ [Cc][ \t]"  to a binary data file which begins with "0x83 0x05" or "0xC3 
0x05"?
Also with different character set encodings, I can not explain it.
Is this a bug or I too stupid?

In my program this behavior is fatal.
But no problem, this is a rare file type and in my program I could catch the 
hits. 
My question is: Is it possible that other file types are similarly affected 
in version 5.04 ?

Thanks,
-Roberto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20101016/5971284d/attachment.html>

From christos at zoulas.com  Fri Oct 22 01:08:51 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 21 Oct 2010 18:08:51 -0400
Subject: [file] small manpage fix
In-Reply-To: <20101021194839.GH19295@acme.spoerlein.net>
	from Ulrich =?utf-8?B?U3DDtnJsZWlu?= (Oct 21,  9:48pm)
Message-ID: <20101021220852.012CD56426@rebar.astron.com>

On Oct 21,  9:48pm, uqs at spoerlein.net (Ulrich =?utf-8?B?U3DDtnJsZWlu?=) wrote:
-- Subject: [file] small manpage fix

| Hello Christos,
| 
| Please find attached a small mdoc fix for the magic.man manpage which
| will make it render correctly on mandoc/mdocml and not change groff
| rendering at all.
| 
| Thanks,
| Uli

You got it!

Thanks,

christos


From jpinon at olfeo.com  Fri Oct 22 17:14:57 2010
From: jpinon at olfeo.com (Julien Pinon)
Date: Fri, 22 Oct 2010 16:14:57 +0200
Subject: Incorrect mime info for windows executables
Message-ID: <83r5fifh5q.fsf@cookie.olfeo-lab.net>

Hi,

I allways  get application/octet-stream  mime type for  some dos/windows
executables with any 5-0x release. For instance with putty.exe
(http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe) :

with libmagic 4-26:

vz-3-12415:/tmp# file putty.exe
putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit

vz-3-12415:/tmp# file -i putty.exe
putty.exe: application/x-dosexec


with libmagic 5-04:

vz-2-12415:/tmp# file putty.exe
putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit

vz-2-12415:/tmp# file -i putty.exe
putty.exe: application/octet-stream; charset=binary


I was  not able to  find if this  is a known problem  or not. Do  I miss
something ?


From cchittleborough+comp.file at cluemail.com  Sun Oct 24 08:28:37 2010
From: cchittleborough+comp.file at cluemail.com (Christopher Chittleborough)
Date: Sun, 24 Oct 2010 15:58:37 +1030
Subject: Patch: correct some comments in file.h
Message-ID: <1287898117.4992.1401629429@webmail.messagingengine.com>

The last few comments in the definition of  struct magic  in file.h seem
to have bit-rotted. I think this fixes them.

--- file-5.04/src/file.h        2010-01-20 12:02:39.000000000 +1030
+++ file-5.04.CWC/src/file.h    2010-06-02 19:45:47.000000000 +0930
@@ -265,11 +265,11 @@
 #define str_flags _u._s._flags
        /* Words 9-16 */
        union VALUETYPE value;  /* either number or string */
-       /* Words 17-24 */
+       /* Words 17-32 */
        char desc[MAXDESC];     /* description */
-       /* Words 25-32 */
+       /* Words 33-48 */
        char mimetype[MAXDESC]; /* MIME type */
-       /* Words 33-34 */
+       /* Words 49-50 */
        char apple[8];
 };


From cchittleborough+comp.file at cluemail.com  Sun Oct 24 08:33:52 2010
From: cchittleborough+comp.file at cluemail.com (Christopher Chittleborough)
Date: Sun, 24 Oct 2010 16:03:52 +1030
Subject: Patch: cope with consecutive spaces in string/W rules
Message-ID: <1287898432.5681.1401629477@webmail.messagingengine.com>

This patch fixes a small problem with string/W: if the rule has a string
value containing multiple consecutive spaces, it would never match
because file_strncmp() in softmagic.c always skips all the spaces in the
object being analyzed when it sees a space in the rule value, meaning
the
second space in the rule will never match.

--- file-5.04.ORIGINAL/src/softmagic.c  2009-10-19 23:40:20+1030
+++ file-5.04/src/softmagic.c   2010-06-03 01:26:31 +0930
@@ -1640,8 +1640,10 @@
                            isspace(*a)) {
                                a++;
                                if (isspace(*b++)) {
-                                       while (isspace(*b))
-                                               b++;
+                                       if (!isspace(*a)) {
+                                               while (isspace(*b))
+                                                       b++;
+                                       }
                                }
                                else {
                                        v = 1;


From cchittleborough+comp.file at cluemail.com  Sun Oct 24 08:48:33 2010
From: cchittleborough+comp.file at cluemail.com (Christopher Chittleborough)
Date: Sun, 24 Oct 2010 16:18:33 +1030
Subject: Patch to magdir/archive: handle more ZIP-based formats
Message-ID: <1287899313.8639.1401629671@webmail.messagingengine.com>

Some recent file formats (eg., OpenDocument, EPUB) are ZIP archives
whose first member must be named 'mimetype' and must contain the
IANA-style MIME type for the format, stored without compression or any
'extra field'.

In file v5.04, ZIPs of this kind with a mimetype other than 'vnd.kde.*',
'vnd.sun.xml.*' and 'vnd.oasis.opendocument.*' are reported as
'data'. This includes .epub ebooks. The following patch fixes that, and
makes some other improvements to the handling of ZIPs. (There's a copy
at http://cchittleborough.cluemail.com/archive-EPUB-etc.patch if anyone
wants it.)

There is an ugly hack in this patch which will cause it to mis-report
unrecognised MIME types that contain 'P'. (Fortunately, most MIME types
are lowercase-only.) When reporting the contents of the 'mimetype'
member, it stops at the first 'P' it finds, on the assumption that such
a 'P' is the start of the header signature (PK\3\4) for the next archive
member.

Which leads me to suggest a non-trivial enhancement to file: a way to
use *.%s or something equivalent in message strings, with the maximum
length coming from an arbitrary integer field in the file being
examined. (Here's a top-of-my-head possibility: we could say that a rule
with a %.*s format must be followed by a rule with an integer type and
no message.) Is anyone interested in adding this sort of complication to
file? Is there a better way to report non-NUL-terminated strings?

Cheers -- Chris Chittleborough


--- file-5.04/magic/Magdir/archive      2009-12-04 15:00:47 +0000
+++ file-5.04.CWC/magic/Magdir/archive  2010-10-24 04:03:00 +0000
@@ -567,8 +567,12 @@
 !:mime  application/zip
 
 # ZIP archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
+#   Some recent formats (eg., ODF) use ZIP archives whose first member
is named
+#   "mimetype" and contains the format's IANA-registered media type.
They match
+#      26 string \x8\0\0\0mimetype
+#   (first member of zip has 8-byte name "mimetype" and no 'extra
field').
 0       string          PK\003\004
->30    ubelong         !0x6d696d65
+>26    string          !\x8\0\0\0mimetype
 >>4	byte		0x00		Zip archive data
 !:mime  application/zip
 >>4	byte		0x09		Zip archive data, at least v0.9 to extract
@@ -582,13 +586,15 @@
 >>4	byte		0x14		Zip archive data, at least v2.0 to extract
 !:mime  application/zip
 
-# OpenOffice.org / KOffice / StarOffice documents
-# Listed here because they ARE zip files
+# KOffice / StarOffice / OpenDocument documents
+# Listed here because they ARE zip files. They start with a member
named
+#  "mimetype" specifying a registered media type.
 #
 # From: Abel Cheung <abel at oaka.org>
->30    string          mimetype
+>26    string          \x8\0\0\0mimetypeapplication/
 
 # KOffice (1.2 or above) formats
+#  mimetype contains 'application/vnd.kde.<SUBTYPE>'
 >>50	string	vnd.kde.		KOffice (>=1.2)
 >>>58	string	karbon			Karbon document
 >>>58	string	kchart			KChart document
@@ -600,6 +606,7 @@
 >>>58	string	kword			KWord document
 
 # OpenOffice formats (for OpenOffice 1.x / StarOffice 6/7)
+#  mimetype contains 'application/vnd.sun.xml.<SUBTYPE>'
 >>50	string	vnd.sun.xml.		OpenOffice.org 1.x
 >>>62	string	writer			Writer
 >>>>68	byte	!0x2e			document
@@ -619,6 +626,7 @@
 
 # OpenDocument formats (for OpenOffice 2.x / StarOffice >= 8)
 # http://lists.oasis-open.org/archives/office/200505/msg00006.html
+#  mimetype contains 'application/vnd.oasis.opendocument.<SUBTYPE>'
 >>50	string	vnd.oasis.opendocument.	OpenDocument
 >>>73	string	text
 >>>>77	byte	!0x2d			Text
@@ -662,6 +670,22 @@
 >>>>78	string	-template		Template
 !:mime  application/vnd.oasis.opendocument.image-template
 
+# EPUB documents (.epub eBooks) are also zip archives starting with a
+# 'mimetype' member, which contains 'application/epub+zip'.
+>>50   string          epub+zip        epub eBook data
+!:mime application/epub+zip
+
+# Catch other ZIP-with-mimetype formats
+>>50           string          !epub+zip
+>>>50          string          !vnd.oasis.opendocument.
+>>>>50         string          !vnd.sun.xml.
+>>>>>50                string          !vnd.kde.
+>>>>>>38       regex           [!-OQ-~]+       Zip data (mimetype
"%s"?)
+>26    string          \x8\0\0\0mimetype
+>>38   string          !application/
+>>>38  regex           [!-OQ-~]+               Zip data (mimetype
"%s"?)
+
+
 # Zoo archiver
 20      lelong          0xfdc4a7dc      Zoo archive data
 !:mime  application/x-zoo


From christos at zoulas.com  Sun Oct 24 17:35:57 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 24 Oct 2010 10:35:57 -0400
Subject: Patch: correct some comments in file.h
In-Reply-To: <1287898117.4992.1401629429@webmail.messagingengine.com>
	from "Christopher Chittleborough" (Oct 24,  3:58pm)
Message-ID: <20101024143557.70B9656427@rebar.astron.com>

On Oct 24,  3:58pm, cchittleborough+comp.file at cluemail.com ("Christopher Chittleborough") wrote:
-- Subject: Patch: correct some comments in file.h

| The last few comments in the definition of  struct magic  in file.h seem
| to have bit-rotted. I think this fixes them.

Thanks a lot.

christos


From christos at zoulas.com  Sun Oct 24 17:50:13 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 24 Oct 2010 10:50:13 -0400
Subject: Patch to magdir/archive: handle more ZIP-based formats
In-Reply-To: <1287899313.8639.1401629671@webmail.messagingengine.com>
	from "Christopher Chittleborough" (Oct 24,  4:18pm)
Message-ID: <20101024145013.3FB9556426@rebar.astron.com>

On Oct 24,  4:18pm, cchittleborough+comp.file at cluemail.com ("Christopher Chittleborough") wrote:
-- Subject: Patch to magdir/archive: handle more ZIP-based formats

| Some recent file formats (eg., OpenDocument, EPUB) are ZIP archives
| whose first member must be named 'mimetype' and must contain the
| IANA-style MIME type for the format, stored without compression or any
| 'extra field'.
| 
| In file v5.04, ZIPs of this kind with a mimetype other than 'vnd.kde.*',
| 'vnd.sun.xml.*' and 'vnd.oasis.opendocument.*' are reported as
| 'data'. This includes .epub ebooks. The following patch fixes that, and
| makes some other improvements to the handling of ZIPs. (There's a copy
| at http://cchittleborough.cluemail.com/archive-EPUB-etc.patch if anyone
| wants it.)
| 
| There is an ugly hack in this patch which will cause it to mis-report
| unrecognised MIME types that contain 'P'. (Fortunately, most MIME types
| are lowercase-only.) When reporting the contents of the 'mimetype'
| member, it stops at the first 'P' it finds, on the assumption that such
| a 'P' is the start of the header signature (PK\3\4) for the next archive
| member.
| 
| Which leads me to suggest a non-trivial enhancement to file: a way to
| use *.%s or something equivalent in message strings, with the maximum
| length coming from an arbitrary integer field in the file being
| examined. (Here's a top-of-my-head possibility: we could say that a rule
| with a %.*s format must be followed by a rule with an integer type and
| no message.) Is anyone interested in adding this sort of complication to
| file? Is there a better way to report non-NUL-terminated strings?

We can add this. I'll think about the syntax.

| Cheers -- Chris Chittleborough

The patch does not apply because someone else has already added support
for OpenDocument. Can you please take a look at the new one and resubmit?

christos


#------------------------------------------------------------------------------
# $File: archive,v 1.58 2010/09/20 19:19:16 rrt Exp $
# archive:  file(1) magic for archive formats (see also "msdos" for self-
#           extracting compressed archives)
#
# cpio, ar, arc, arj, hpack, lha/lharc, rar, squish, uc2, zip, zoo, etc.
# pre-POSIX "tar" archives are handled in the C code.

# POSIX tar archives
257	string		ustar\0		POSIX tar archive
!:mime	application/x-tar # encoding: posix
257	string		ustar\040\040\0	GNU tar archive
!:mime	application/x-tar # encoding: gnu

# cpio archives
#
# Yes, the top two "cpio archive" formats *are* supposed to just be "short".
# The idea is to indicate archives produced on machines with the same
# byte order as the machine running "file" with "cpio archive", and
# to indicate archives produced on machines with the opposite byte order
# from the machine running "file" with "byte-swapped cpio archive".
#
# The SVR4 "cpio(4)" hints that there are additional formats, but they
# are defined as "short"s; I think all the new formats are
# character-header formats and thus are strings, not numbers.
0	short		070707		cpio archive
!:mime	application/x-cpio
0	short		0143561		byte-swapped cpio archive
!:mime	application/x-cpio # encoding: swapped
0	string		070707		ASCII cpio archive (pre-SVR4 or odc)
0	string		070701		ASCII cpio archive (SVR4 with no CRC)
0	string		070702		ASCII cpio archive (SVR4 with CRC)

# Debian package (needs to go before regular portable archives)
#
0	string		=!<arch>\ndebian
!:mime	application/x-debian-package
>8	string		debian-split	part of multipart Debian package
>8	string		debian-binary	Debian binary package
>8	string		!debian
>68	string		>\0		(format %s)
# These next two lines do not work, because a bzip2 Debian archive
# still uses gzip for the control.tar (first in the archive).  Only
# data.tar varies, and the location of its filename varies too.
# file/libmagic does not current have support for ascii-string based
# (offsets) as of 2005-09-15.
#>81	string		bz2		\b, uses bzip2 compression
#>84	string		gz		\b, uses gzip compression
#>136	ledate		x		created: %s

# other archives
0	long		0177555		very old archive
0	short		0177555		very old PDP-11 archive
0	long		0177545		old archive
0	short		0177545		old PDP-11 archive
0	long		0100554		apl workspace
0	string		=<ar>		archive
!:mime	application/x-archive

# MIPS archive (needs to go before regular portable archives)
#
0	string	=!<arch>\n__________E	MIPS archive
>20	string	U			with MIPS Ucode members
>21	string	L			with MIPSEL members
>21	string	B			with MIPSEB members
>19	string	L			and an EL hash table
>19	string	B			and an EB hash table
>22	string	X			-- out of date

0	search/1	-h-		Software Tools format archive text

#
# XXX - why are there multiple <ar> thingies?  Note that 0x213c6172 is
# "!<ar", so, for new-style (4.xBSD/SVR2andup) archives, we have:
#
# 0	string		=!<arch>		current ar archive
# 0	long		0x213c6172	archive file
#
# and for SVR1 archives, we have:
#
# 0	string		\<ar>		System V Release 1 ar archive
# 0	string		=<ar>		archive
#
# XXX - did Aegis really store shared libraries, breakpointed modules,
# and absolute code program modules in the same format as new-style
# "ar" archives?
#
0	string		=!<arch>		current ar archive
!:mime	application/x-archive
>8	string		__.SYMDEF	random library
>0	belong		=65538		- pre SR9.5
>0	belong		=65539		- post SR9.5
>0	beshort		2		- object archive
>0	beshort		3		- shared library module
>0	beshort		4		- debug break-pointed module
>0	beshort		5		- absolute code program module
0	string		\<ar>		System V Release 1 ar archive
0	string		=<ar>		archive
#
# XXX - from "vax", which appears to collect a bunch of byte-swapped
# thingies, to help you recognize VAX files on big-endian machines;
# with "leshort", "lelong", and "string", that's no longer necessary....
#
0	belong		0x65ff0000	VAX 3.0 archive
0	belong		0x3c61723e	VAX 5.0 archive
#
0	long		0x213c6172	archive file
0	lelong		0177555		very old VAX archive
0	leshort		0177555		very old PDP-11 archive
#
# XXX - "pdp" claims that 0177545 can have an __.SYMDEF member and thus
# be a random library (it said 0xff65 rather than 0177545).
#
0	lelong		0177545		old VAX archive
>8	string		__.SYMDEF	random library
0	leshort		0177545		old PDP-11 archive
>8	string		__.SYMDEF	random library
#
# From "pdp" (but why a 4-byte quantity?)
#
0	lelong		0x39bed		PDP-11 old archive
0	lelong		0x39bee		PDP-11 4.0 archive

# ARC archiver, from Daniel Quinlan (quinlan at yggdrasil.com)
#
# The first byte is the magic (0x1a), byte 2 is the compression type for
# the first file (0x01 through 0x09), and bytes 3 to 15 are the MS-DOS
# filename of the first file (null terminated).  Since some types collide
# we only test some types on basis of frequency: 0x08 (83%), 0x09 (5%),
# 0x02 (5%), 0x03 (3%), 0x04 (2%), 0x06 (2%).  0x01 collides with terminfo.
0	lelong&0x8080ffff	0x0000081a	ARC archive data, dynamic LZW
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000091a	ARC archive data, squashed
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000021a	ARC archive data, uncompressed
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000031a	ARC archive data, packed
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000041a	ARC archive data, squeezed
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000061a	ARC archive data, crunched
!:mime	application/x-arc
# [JW] stuff taken from idarc, obviously ARC successors:
0	lelong&0x8080ffff	0x00000a1a	PAK archive data
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000141a	ARC+ archive data
!:mime	application/x-arc
0	lelong&0x8080ffff	0x0000481a	HYP archive data
!:mime	application/x-arc

# Acorn archive formats (Disaster prone simpleton, m91dps at ecs.ox.ac.uk)
# I can't create either SPARK or ArcFS archives so I have not tested this stuff
# [GRR:  the original entries collide with ARC, above; replaced with combined
#  version (not tested)]
#0	byte		0x1a		RISC OS archive (spark format)
0	string		\032archive	RISC OS archive (ArcFS format)
0       string          Archive\000     RISC OS archive (ArcFS format)

# All these were taken from idarc, many could not be verified. Unfortunately,
# there were many low-quality sigs, i.e. easy to trigger false positives.
# Please notify me of any real-world fishy/ambiguous signatures and I'll try
# to get my hands on the actual archiver and see if I find something better. [JW]
# probably many can be enhanced by finding some 0-byte or control char near the start

# idarc calls this Crush/Uncompressed... *shrug*
0	string	CRUSH Crush archive data
# Squeeze It (.sqz)
0	string	HLSQZ Squeeze It archive data
# SQWEZ
0	string	SQWEZ SQWEZ archive data
# HPack (.hpk)
0	string	HPAK HPack archive data
# HAP
0	string	\x91\x33HF HAP archive data
# MD/MDCD
0	string	MDmd MDCD archive data
# LIM
0	string	LIM\x1a LIM archive data
# SAR
3	string	LH5 SAR archive data
# BSArc/BS2
0	string	\212\3SB \0 BSArc/BS2 archive data
# MAR
2	string	=-ah MAR archive data
# ACB
0	belong&0x00f800ff	0x00800000 ACB archive data
# CPZ
# TODO, this is what idarc says: 0	string	\0\0\0 CPZ archive data
# JRC
0	string	JRchive JRC archive data
# Quantum
0	string	DS\0 Quantum archive data
# ReSOF
0	string	PK\3\6 ReSOF archive data
# QuArk
0	string	7\4 QuArk archive data
# YAC
14	string	YC YAC archive data
# X1
0	string	X1 X1 archive data
0	string	XhDr X1 archive data
# CDC Codec (.dqt)
0	belong&0xffffe000	0x76ff2000 CDC Codec archive data
# AMGC
0	string	\xad6" AMGC archive data
# NuLIB
0	string	N??F??l?? NuLIB archive data
# PakLeo
0	string	LEOLZW PAKLeo archive data
# ChArc
0	string	SChF ChArc archive data
# PSA
0	string	PSA PSA archive data
# CrossePAC
0	string	DSIGDCC CrossePAC archive data
# Freeze
0	string	\x1f\x9f\x4a\x10\x0a Freeze archive data
# KBoom
0	string	??MP?? KBoom archive data
# NSQ, must go after CDC Codec
0	string	\x76\xff NSQ archive data
# DPA
0	string	Dirk\ Paehl DPA archive data
# BA
# TODO: idarc says "bytes 0-2 == bytes 3-5"
# TTComp
0	string	\0\6 TTComp archive data
# ESP, could this conflict with Easy Software Products' (e.g.ESP ghostscript) documentation?
0	string	ESP ESP archive data
# ZPack
0	string	\1ZPK\1 ZPack archive data
# Sky
0	string	\xbc\x40 Sky archive data
# UFA
0	string	UFA UFA archive data
# Dry
0	string	=-H2O DRY archive data
# FoxSQZ
0	string	FOXSQZ FoxSQZ archive data
# AR7
0	string	,AR7 AR7 archive data
# PPMZ
0	string	PPMZ PPMZ archive data
# MS Compress
4	string	\x88\xf0\x27 MS Compress archive data
# updated by Joerg Jenderek
>9	string	\0
>>0	string	KWAJ
>>>7	string	\321\003	MS Compress archive data
>>>>14	ulong	>0		\b, original size: %ld bytes
>>>>18		ubyte	>0x65
>>>>>18		string	x       \b, was %.8s
>>>>>(10.b-4)	string	x       \b.%.3s
# MP3 (archiver, not lossy audio compression)
0	string	MP3\x1a MP3-Archiver archive data
# ZET
0	string	OZ?? ZET archive data
# TSComp
0	string	\x65\x5d\x13\x8c\x08\x01\x03\x00 TSComp archive data
# ARQ
0	string	gW\4\1 ARQ archive data
# Squash
3	string	OctSqu Squash archive data
# Terse
0	string	\5\1\1\0 Terse archive data
# PUCrunch
0	string	\x01\x08\x0b\x08\xef\x00\x9e\x32\x30\x36\x31 PUCrunch archive data
# UHarc
0	string	UHA UHarc archive data
# ABComp
0	string	\2AB ABComp archive data
0	string	\3AB2 ABComp archive data
# CMP
0	string	CO\0 CMP archive data
# Splint
0	string	\x93\xb9\x06 Splint archive data
# InstallShield
0	string   \x13\x5d\x65\x8c InstallShield Z archive Data
# Gather
1	string	GTH Gather archive data
# BOA
0	string	BOA BOA archive data
# RAX
0	string	ULEB\xa RAX archive data
# Xtreme
0	string	ULEB\0 Xtreme archive data
# Pack Magic
0	string	@??\1\0 Pack Magic archive data
# BTS
0	belong&0xfeffffff	0x1a034465 BTS archive data
# ELI 5750
0	string	Ora\  ELI 5750 archive data
# QFC
0	string	\x1aFC\x1a QFC archive data
0	string	\x1aQF\x1a QFC archive data
# PRO-PACK
0	string	RNC PRO-PACK archive data
# 777
0	string	777 777 archive data
# LZS221
0	string	sTaC LZS221 archive data
# HPA
0	string	HPA HPA archive data
# Arhangel
0	string	LG Arhangel archive data
# EXP1, uses bzip2
0	string	0123456789012345BZh EXP1 archive data
# IMP
0	string	IMP\xa IMP archive data
# NRV
0	string	\x00\x9E\x6E\x72\x76\xFF NRV archive data
# Squish
0	string	\x73\xb2\x90\xf4 Squish archive data
# Par
0	string	PHILIPP Par archive data
0	string	PAR Par archive data
# HIT
0	string	UB HIT archive data
# SBX
0	belong&0xfffff000	0x53423000 SBX archive data
# NaShrink
0	string	NSK NaShrink archive data
# SAPCAR
0	string	#\ CAR\ archive\ header SAPCAR archive data
0	string	CAR\ 2.00RG SAPCAR archive data
# Disintegrator
0	string	DST Disintegrator archive data
# ASD
0	string	ASD ASD archive data
# InstallShield CAB
0	string	ISc( InstallShield CAB
# TOP4
0	string	T4\x1a TOP4 archive data
# BatComp left out: sig looks like COM executable
# so TODO: get real 4dos batcomp file and find sig
# BlakHole
0	string	BH\5\7 BlakHole archive data
# BIX
0	string	BIX0 BIX archive data
# ChiefLZA
0	string	ChfLZ ChiefLZA archive data
# Blink
0	string	Blink Blink archive data
# Logitech Compress
0	string	\xda\xfa Logitech Compress archive data
# ARS-Sfx (FIXME: really a SFX? then goto COM/EXE)
1	string	(C)\ STEPANYUK ARS-Sfx archive data
# AKT/AKT32
0	string	AKT32 AKT32 archive data
0	string	AKT AKT archive data
# NPack
0	string	MSTSM NPack archive data
# PFT
0	string	\0\x50\0\x14 PFT archive data
# SemOne
0	string	SEM SemOne archive data
# PPMD
0	string	\x8f\xaf\xac\x84 PPMD archive data
# FIZ
0	string	FIZ FIZ archive data
# MSXiE
0	belong&0xfffff0f0	0x4d530000 MSXiE archive data
# DeepFreezer
0	belong&0xfffffff0	0x797a3030 DeepFreezer archive data
# DC
0	string	=<DC- DC archive data
# TPac
0	string	\4TPAC\3 TPac archive data
# Ai
0	string	Ai\1\1\0 Ai archive data
0	string	Ai\1\0\0 Ai archive data
# Ai32
0	string	Ai\2\0 Ai32 archive data
0	string	Ai\2\1 Ai32 archive data
# SBC
0	string	SBC SBC archive data
# Ybs
0	string	YBS Ybs archive data
# DitPack
0	string	\x9e\0\0 DitPack archive data
# DMS
0	string	DMS! DMS archive data
# EPC
0	string	\x8f\xaf\xac\x8c EPC archive data
# VSARC
0	string	VS\x1a VSARC archive data
# PDZ
0	string	PDZ PDZ archive data
# ReDuq
0	string	rdqx ReDuq archive data
# GCA
0	string	GCAX GCA archive data
# PPMN
0	string	pN PPMN archive data
# WinImage
3	string	WINIMAGE WinImage archive data
# Compressia
0	string	CMP0CMP Compressia archive data
# UHBC
0	string	UHB UHBC archive data
# WinHKI
0	string	\x61\x5C\x04\x05 WinHKI archive data
# WWPack data file
0	string	WWP WWPack archive data
# BSN (BSA, PTS-DOS)
0	string	\xffBSG BSN archive data
1	string	\xffBSG BSN archive data
3	string	\xffBSG BSN archive data
1	string	\0\xae\2 BSN archive data
1	string	\0\xae\3 BSN archive data
1	string	\0\xae\7 BSN archive data
# AIN
0	string	\x33\x18 AIN archive data
0	string	\x33\x17 AIN archive data
# XPA32
0	string	xpa\0\1 XPA32 archive data
# SZip (TODO: doesn't catch all versions)
0	string	SZ\x0a\4 SZip archive data
# XPack DiskImage
0	string	jm XPack DiskImage archive data
# XPack Data
0	string	xpa XPack archive data
# XPack Single Data
0	string	??\ jm XPack single archive data

# TODO: missing due to unknown magic/magic at end of file:
#DWC
#ARG
#ZAR
#PC/3270
#InstallIt
#RKive
#RK
#XPack Diskimage

# These were inspired by idarc, but actually verified
# Dzip archiver (.dz)
0	string	DZ Dzip archive data
>2	byte	x \b, version %i
>3	byte	x \b.%i
# ZZip archiver (.zz)
0	string	ZZ\ \0\0 ZZip archive data
0	string	ZZ0 ZZip archive data
# PAQ archiver (.paq)
0	string	\xaa\x40\x5f\x77\x1f\xe5\x82\x0d PAQ archive data
0	string	PAQ PAQ archive data
>3	byte&0xf0	0x30
>>3	byte	x (v%c)
# JAR archiver (.j), this is the successor to ARJ, not Java's JAR (which is essentially ZIP)
0xe	string	\x1aJar\x1b JAR (ARJ Software, Inc.) archive data
0	string	JARCS JAR (ARJ Software, Inc.) archive data

# ARJ archiver (jason at jarthur.Claremont.EDU)
0	leshort		0xea60		ARJ archive data
!:mime	application/x-arj
>5	byte		x		\b, v%d,
>8	byte		&0x04		multi-volume,
>8	byte		&0x10		slash-switched,
>8	byte		&0x20		backup,
>34	string		x		original name: %s,
>7	byte		0		os: MS-DOS
>7	byte		1		os: PRIMOS
>7	byte		2		os: Unix
>7	byte		3		os: Amiga
>7	byte		4		os: Macintosh
>7	byte		5		os: OS/2
>7	byte		6		os: Apple ][ GS
>7	byte		7		os: Atari ST
>7	byte		8		os: NeXT
>7	byte		9		os: VAX/VMS
>3	byte		>0		%d]
# [JW] idarc says this is also possible
2	leshort		0xea60		ARJ archive data

# HA archiver (Greg Roelofs, newt at uchicago.edu)
# This is a really bad format. A file containing HAWAII will match this...
#0	string		HA		HA archive data,
#>2	leshort		=1		1 file,
#>2	leshort		>1		%u files,
#>4	byte&0x0f	=0		first is type CPY
#>4	byte&0x0f	=1		first is type ASC
#>4	byte&0x0f	=2		first is type HSC
#>4	byte&0x0f	=0x0e		first is type DIR
#>4	byte&0x0f	=0x0f		first is type SPECIAL
# suggestion: at least identify small archives (<1024 files)
0  belong&0xffff00fc 0x48410000 HA archive data
>2	leshort		=1		1 file,
>2	leshort		>1		%u files,
>4	byte&0x0f	=0		first is type CPY
>4	byte&0x0f	=1		first is type ASC
>4	byte&0x0f	=2		first is type HSC
>4	byte&0x0f	=0x0e		first is type DIR
>4	byte&0x0f	=0x0f		first is type SPECIAL

# HPACK archiver (Peter Gutmann, pgut1 at cs.aukuni.ac.nz)
0	string		HPAK		HPACK archive data

# JAM Archive volume format, by Dmitry.Kohmanyuk at UA.net
0	string		\351,\001JAM\           JAM archive,
>7	string		>\0			version %.4s
>0x26	byte		=0x27			-
>>0x2b	string          >\0			label %.11s,
>>0x27	lelong		x			serial %08x,
>>0x36	string		>\0			fstype %.8s

# LHARC/LHA archiver (Greg Roelofs, newt at uchicago.edu)
2	string		-lh0-		LHarc 1.x/ARX archive data [lh0]
!:mime	application/x-lharc
2	string		-lh1-		LHarc 1.x/ARX archive data [lh1]
!:mime	application/x-lharc
2	string		-lz4-		LHarc 1.x archive data [lz4]
!:mime	application/x-lharc
2	string		-lz5-		LHarc 1.x archive data [lz5]
!:mime	application/x-lharc
#	[never seen any but the last; -lh4- reported in comp.compression:]
2	string		-lzs-		LHa/LZS archive data [lzs]
!:mime	application/x-lha
2	string		-lh\40-		LHa 2.x? archive data [lh ]
!:mime	application/x-lha
2	string		-lhd-		LHa 2.x? archive data [lhd]
!:mime	application/x-lha
2	string		-lh2-		LHa 2.x? archive data [lh2]
!:mime	application/x-lha
2	string		-lh3-		LHa 2.x? archive data [lh3]
!:mime	application/x-lha
2	string		-lh4-		LHa (2.x) archive data [lh4]
!:mime	application/x-lha
2	string		-lh5-		LHa (2.x) archive data [lh5]
!:mime	application/x-lha
2	string		-lh6-		LHa (2.x) archive data [lh6]
!:mime	application/x-lha
2	string		-lh7-		LHa (2.x)/LHark archive data [lh7]
!:mime	application/x-lha
>20	byte		x		- header level %d
# taken from idarc [JW]
2   string      -lZ         PUT archive data
2   string      -lz         LZS archive data
2   string      -sw1-       Swag archive data

# RAR archiver (Greg Roelofs, newt at uchicago.edu)
0	string		Rar!		RAR archive data,
!:mime	application/x-rar
>44	byte		x		v%0x,
>10	byte		>0		flags:
>>10	byte		&0x01		Archive volume,
>>10	byte		&0x02		Commented,
>>10	byte		&0x04		Locked,
>>10	byte		&0x08		Solid,
>>10	byte		&0x20		Authenticated,
>35	byte		0		os: MS-DOS
>35	byte		1		os: OS/2
>35	byte		2		os: Win32
>35	byte		3		os: Unix
# some old version? idarc says:
0   string      RE\x7e\x5e  RAR archive data

# SQUISH archiver (Greg Roelofs, newt at uchicago.edu)
0	string		SQSH		squished archive data (Acorn RISCOS)

# UC2 archiver (Greg Roelofs, newt at uchicago.edu)
# [JW] see exe section for self-extracting version
0	string		UC2\x1a		UC2 archive data

# PKZIP multi-volume archive
0	string		PK\x07\x08PK\x03\x04	Zip multi-volume archive data, at least PKZIP v2.50 to extract
!:mime	application/zip

# Zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
0	string		PK\003\004

# Specialised zip formats
>30	string		mimetype

# OpenOffice.org / KOffice / StarOffice documents
# From: Abel Cheung <abel at oaka.org>

# KOffice (1.2 or above) formats
>>50	string	vnd.kde.		KOffice (>=1.2)
>>>58	string	karbon			Karbon document
>>>58	string	kchart			KChart document
>>>58	string	kformula		KFormula document
>>>58	string	kivio			Kivio document
>>>58	string	kontour			Kontour document
>>>58	string	kpresenter		KPresenter document
>>>58	string	kspread			KSpread document
>>>58	string	kword			KWord document

# OpenOffice formats (for OpenOffice 1.x / StarOffice 6/7)
#  mimetype contains 'application/vnd.sun.xml.<SUBTYPE>'
>>50	string	vnd.sun.xml.		OpenOffice.org 1.x
>>>62	string	writer			Writer
>>>>68	byte	!0x2e			document
>>>>68	string	.template		template
>>>>68	string	.global			global document
>>>62	string	calc			Calc
>>>>66	byte	!0x2e			spreadsheet
>>>>66	string	.template		template
>>>62	string	draw			Draw
>>>>66	byte	!0x2e			document
>>>>66	string	.template		template
>>>62	string	impress			Impress
>>>>69	byte	!0x2e			presentation
>>>>69	string	.template		template
>>>62	string	math			Math document
>>>62	string	base			Database file

# OpenDocument formats (for OpenOffice 2.x / StarOffice >= 8)
# http://lists.oasis-open.org/archives/office/200505/msg00006.html
#  mimetype contains 'application/vnd.oasis.opendocument.<SUBTYPE>'
>>50	string	vnd.oasis.opendocument.	OpenDocument
>>>73	string	text
>>>>77	byte	!0x2d			Text
!:mime	application/vnd.oasis.opendocument.text
>>>>77	string	-template		Text Template
!:mime	application/vnd.oasis.opendocument.text-template
>>>>77	string	-web			HTML Document Template
!:mime	application/vnd.oasis.opendocument.text-web
>>>>77	string	-master			Master Document
!:mime	application/vnd.oasis.opendocument.text-master
>>>73	string	graphics
>>>>81	byte	!0x2d			Drawing
!:mime	application/vnd.oasis.opendocument.graphics
>>>>81	string	-template		Template
!:mime	application/vnd.oasis.opendocument.graphics-template
>>>73	string	presentation
>>>>85	byte	!0x2d			Presentation
!:mime	application/vnd.oasis.opendocument.presentation
>>>>85	string	-template		Template
!:mime	application/vnd.oasis.opendocument.presentation-template
>>>73	string	spreadsheet
>>>>84	byte	!0x2d			Spreadsheet
!:mime	application/vnd.oasis.opendocument.spreadsheet
>>>>84	string	-template		Template
!:mime	application/vnd.oasis.opendocument.spreadsheet-template
>>>73	string	chart
>>>>78	byte	!0x2d			Chart
!:mime	application/vnd.oasis.opendocument.chart
>>>>78	string	-template		Template
!:mime	application/vnd.oasis.opendocument.chart-template
>>>73	string	formula
>>>>80	byte	!0x2d			Formula
!:mime	application/vnd.oasis.opendocument.formula
>>>>80	string	-template		Template
!:mime	application/vnd.oasis.opendocument.formula-template
>>>73	string	database		Database
!:mime	application/vnd.oasis.opendocument.database
>>>73	string	image
>>>>78	byte	!0x2d			Image
!:mime	application/vnd.oasis.opendocument.image
>>>>78	string	-template		Template
!:mime	application/vnd.oasis.opendocument.image-template

# OEBPS Container Format (OCF) 1.0
>>50	string	epub+zip		OEBPS Container Format (OCF) 1.0 (EPUB)
!:mime	application/epub+zip

# Generic zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
# N.B. Must come after specialised formats
# Next line simply matches anything, including "mimetype" as used above
>30	byte		x
>>4	byte		0x00		Zip archive data
!:mime	application/zip
>>4	byte		0x09		Zip archive data, at least v0.9 to extract
!:mime	application/zip
>>4	byte		0x0a		Zip archive data, at least v1.0 to extract
!:mime	application/zip
>>4	byte		0x0b		Zip archive data, at least v1.1 to extract
!:mime	application/zip
>>0x161	string		WINZIP          Zip archive data, WinZIP self-extracting
!:mime	application/zip
>>4	byte		0x14		Zip archive data, at least v2.0 to extract
!:mime	application/zip

# EPUB (OEBPS) books using OCF (Open Container Format)
# From: Adam Buchbinder <adam.buchbinder at gmail.com>
# URL: http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm, section 4.
>>38	string	application/epub+zip	EPUB ebook
!:mime	application/epub+zip

# EPUB documents (.epub eBooks) are also zip archives starting with a
# 'mimetype' member, which contains 'application/epub+zip'.
>>50   string          epub+zip        epub eBook data
!:mime application/epub+zip

# Catch other ZIP-with-mimetype formats
>>50           string          !epub+zip
>>>50          string          !vnd.oasis.opendocument.
>>>>50         string          !vnd.sun.xml.
>>>>>50                string          !vnd.kde.
>>>>>>38       regex           [!-OQ-~]+       Zip data (mimetype "%s"?)
>26    string          \x8\0\0\0mimetype
>>38   string          !application/
>>>38  regex           [!-OQ-~]+               Zip data (mimetype "%s"?)


# Zoo archiver
20	lelong		0xfdc4a7dc	Zoo archive data
!:mime	application/x-zoo
>4	byte		>48		\b, v%c.
>>6	byte		>47		\b%c
>>>7	byte		>47		\b%c
>32	byte		>0		\b, modify: v%d
>>33	byte		x		\b.%d+
>42	lelong		0xfdc4a7dc	\b,
>>70	byte		>0		extract: v%d
>>>71	byte		x		\b.%d+

# Shell archives
10	string		#\ This\ is\ a\ shell\ archive	shell archive text
!:mime	application/octet-stream

#
# LBR. NB: May conflict with the questionable
#          "binary Computer Graphics Metafile" format.
#
0       string  \0\ \ \ \ \ \ \ \ \ \ \ \0\0    LBR archive data
#
# PMA (CP/M derivative of LHA)
#
2       string          -pm0-           PMarc archive data [pm0]
2       string          -pm1-           PMarc archive data [pm1]
2       string          -pm2-           PMarc archive data [pm2]
2       string          -pms-           PMarc SFX archive (CP/M, DOS)
5       string          -pc1-           PopCom compressed executable (CP/M)

# From Rafael Laboissiere <rafael at laboissiere.net>
# The Project Revision Control System (see
# http://prcs.sourceforge.net) generates a packaged project
# file which is recognized by the following entry:
0	leshort		0xeb81	PRCS packaged project

# Microsoft cabinets
# by David Necas (Yeti) <yeti at physics.muni.cz>
#0	string	MSCF\0\0\0\0	Microsoft cabinet file data,
#>25	byte	x		v%d
#>24	byte	x		\b.%d
# MPi: All CABs have version 1.3, so this is pointless.
# Better magic in debian-additions.

# GTKtalog catalogs
# by David Necas (Yeti) <yeti at physics.muni.cz>
4	string	gtktalog\       GTKtalog catalog data,
>13	string	3		version 3
>>14	beshort	0x677a		(gzipped)
>>14	beshort	!0x677a		(not gzipped)
>13	string	>3		version %s

############################################################################
# Parity archive reconstruction file, the 'par' file format now used on Usenet.
0       string          PAR\0	PARity archive data
>48	leshort		=0	- Index file
>48	leshort		>0	- file number %d

# Felix von Leitner <felix-file at fefe.de>
0	string	d8:announce	BitTorrent file
!:mime	application/x-bittorrent

# Atari MSA archive - Teemu Hukkanen <tjhukkan at iki.fi>
0       beshort 0x0e0f          Atari MSA archive data
>2      beshort x               \b, %d sectors per track
>4      beshort 0               \b, 1 sided
>4      beshort 1               \b, 2 sided
>6      beshort x               \b, starting track: %d
>8      beshort x               \b, ending track: %d

# Alternate ZIP string (amc at arwen.cs.berkeley.edu)
0	string	PK00PK\003\004	Zip archive data

# ACE archive (from http://www.wotsit.org/download.asp?f=ace)
# by Stefan `Sec` Zehl <sec at 42.org>
7	string		**ACE**		ACE archive data
>15	byte	>0		version %d
>16	byte	=0x00		\b, from MS-DOS
>16	byte	=0x01		\b, from OS/2
>16	byte	=0x02		\b, from Win/32
>16	byte	=0x03		\b, from Unix
>16	byte	=0x04		\b, from MacOS
>16	byte	=0x05		\b, from WinNT
>16	byte	=0x06		\b, from Primos
>16	byte	=0x07		\b, from AppleGS
>16	byte	=0x08		\b, from Atari
>16	byte	=0x09		\b, from Vax/VMS
>16	byte	=0x0A		\b, from Amiga
>16	byte	=0x0B		\b, from Next
>14	byte	x		\b, version %d to extract
>5	leshort &0x0080		\b, multiple volumes,
>>17	byte	x		\b (part %d),
>5	leshort &0x0002		\b, contains comment
>5	leshort	&0x0200		\b, sfx
>5	leshort	&0x0400		\b, small dictionary
>5	leshort	&0x0800		\b, multi-volume
>5	leshort	&0x1000		\b, contains AV-String
>>30	string	\x16*UNREGISTERED\x20VERSION*	(unregistered)
>5	leshort &0x2000		\b, with recovery record
>5	leshort &0x4000		\b, locked
>5	leshort &0x8000		\b, solid
# Date in MS-DOS format (whatever that is)
#>18	lelong	x		Created on

# sfArk : compression program for Soundfonts (sf2) by Dirk Jagdmann
# <doj at cubic.org>
0x1A	string	sfArk		sfArk compressed Soundfont
>0x15	string	2
>>0x1	string	>\0		Version %s
>>0x2A	string	>\0		: %s

# DR-DOS 7.03 Packed File *.??_
0	string	Packed\ File\   Personal NetWare Packed File
>12	string	x               \b, was "%.12s"

# EET archive
# From: Tilman Sauerbeck <tilman at code-monkey.de>
0	belong	0x1ee7ff00	EET archive
!:mime	application/x-eet

# rzip archives
0	string	RZIP		rzip compressed data
>4	byte	x		- version %d
>5	byte	x		\b.%d
>6	belong	x		(%d bytes)

# From: "Robert Dale" <robdale at gmail.com>
0	belong	123		dar archive,
>4	belong	x		label "%.8x
>>8	belong	x		%.8x
>>>12	beshort	x		%.4x"
>14	byte	0x54		end slice
>14	beshort	0x4e4e		multi-part
>14	beshort	0x4e53		multi-part, with -S

# Symbian installation files
#  http://www.thouky.co.uk/software/psifs/sis.html
#  http://developer.symbian.com/main/downloads/papers/SymbianOSv91/softwareinstallsis.pdf
8	lelong	0x10000419	Symbian installation file
!:mime	application/vnd.symbian.install
>4	lelong	0x1000006D	(EPOC release 3/4/5)
>4	lelong	0x10003A12	(EPOC release 6)
0	lelong	0x10201A7A	Symbian installation file (Symbian OS 9.x)
!:mime	x-epoc/x-sisx-app

# From "Nelson A. de Oliveira" <naoliv at gmail.com>
0	string	MPQ\032		MoPaQ (MPQ) archive

# From: Dirk Jagdmann <doj at cubic.org>
# xar archive format: http://code.google.com/p/xar/
0	string	xar!		xar archive
>6	beshort	x		- version %ld

# From: "Nelson A. de Oliveira" <naoliv at gmail.com>
# .kgb
0	string KGB_arch		KGB Archiver file
>10	string x		with compression level %.1s

# xar (eXtensible ARchiver) archive
# From: "David Remahl" <dremahl at apple.com>
0	string	xar!		xar archive
#>4	beshort	x		header size %d
>6	beshort	x		version %d,
#>8	quad	x		compressed TOC: %d,
#>16	quad	x		uncompressed TOC: %d,
>24	belong	0		no checksum
>24	belong	1		SHA-1 checksum
>24	belong	2		MD5 checksum

# Type: Parity Archive
# From: Daniel van Eeden <daniel_e at dds.nl>
0	string	PAR2		Parity Archive Volume Set

# Bacula volume format. (Volumes always start with a block header.)
# URL: http://bacula.org/3.0.x-manuals/en/developers/developers/Block_Header.html
# From: Adam Buchbinder <adam.buchbinder at gmail.com>
12	string	BB02		Bacula volume
>20	bedate	x		\b, started %s


From ghelmer at palisadesys.com  Tue Oct 26 00:17:02 2010
From: ghelmer at palisadesys.com (Guy Helmer)
Date: Mon, 25 Oct 2010 16:17:02 -0500
Subject: Incorrect mime info for windows executables
In-Reply-To: <83r5fifh5q.fsf@cookie.olfeo-lab.net>
References: <83r5fifh5q.fsf@cookie.olfeo-lab.net>
Message-ID: <0784D725-E328-4942-A092-BE5C549723FD@palisadesys.com>

On Oct 22, 2010, at 9:14 AM, Julien Pinon wrote:

> Hi,
> 
> I allways  get application/octet-stream  mime type for  some dos/windows
> executables with any 5-0x release. For instance with putty.exe
> (http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe) :
> 
> with libmagic 4-26:
> 
> vz-3-12415:/tmp# file putty.exe
> putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
> 
> vz-3-12415:/tmp# file -i putty.exe
> putty.exe: application/x-dosexec
> 
> 
> with libmagic 5-04:
> 
> vz-2-12415:/tmp# file putty.exe
> putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
> 
> vz-2-12415:/tmp# file -i putty.exe
> putty.exe: application/octet-stream; charset=binary
> 
> 
> I was  not able to  find if this  is a known problem  or not. Do  I miss
> something ?
> 


This is the change I am using (against the magic files from 5.03) for DOS/Windows executables.

Guy

Index: msdos
===================================================================
--- msdos	(revision 15832)
+++ msdos	(revision 15833)
@@ -52,23 +52,41 @@
 0	string	MZ
 !:mime	application/x-dosexec
 >0x18  leshort <0x40 MS-DOS executable
+!:mime	application/x-dosexec
 >0 string MZ\0\0\0\0\0\0\0\0\0\0PE\0\0 \b, PE for MS Windows
+!:mime	application/x-dosexec
 >>&18	leshort&0x2000	>0	(DLL)
+!:mime	application/x-dosexec
 >>&88	leshort		0	(unknown subsystem)
+!:mime	application/x-dosexec
 >>&88	leshort		1	(native)
+!:mime	application/x-dosexec
 >>&88	leshort		2	(GUI)
+!:mime	application/x-dosexec
 >>&88	leshort		3	(console)
+!:mime	application/x-dosexec
 >>&88	leshort		7	(POSIX)
+!:mime	application/x-dosexec
 >>&0	leshort		0x0	unknown processor
+!:mime	application/x-dosexec
 >>&0	leshort		0x14c	Intel 80386
+!:mime	application/x-dosexec
 >>&0	leshort		0x166	MIPS R4000
+!:mime	application/x-dosexec
 >>&0	leshort		0x184	Alpha
+!:mime	application/x-dosexec
 >>&0	leshort		0x268	Motorola 68000
+!:mime	application/x-dosexec
 >>&0	leshort		0x1f0	PowerPC
+!:mime	application/x-dosexec
 >>&0	leshort		0x290	PA-RISC
+!:mime	application/x-dosexec
 >>&18	leshort&0x0100	>0	32-bit
+!:mime	application/x-dosexec
 >>&18	leshort&0x1000	>0	system file
+!:mime	application/x-dosexec
 >>&0xf4 search/0x140 \x0\x40\x1\x0
+!:mime	application/x-dosexec
 >>>(&0.l+(4)) string MSCF \b, WinHKI CAB self-extracting archive
 >30		string	Copyright\ 1989-1990\ PKWARE\ Inc.	Self-extracting PKZIP archive
 !:mime	application/zip
@@ -79,17 +97,23 @@
 >0x18  leshort >0x3f
 >>(0x3c.l) string PE\0\0 PE
 >>>(0x3c.l+25) byte		1 \b32 executable
+!:mime	application/x-dosexec
 >>>(0x3c.l+25) byte		2 \b32+ executable
+!:mime	application/x-dosexec
 # hooray, there's a DOS extender using the PE format, with a valid PE
 # executable inside (which just prints a message and exits if run in win)
 >>>(0x3c.l+92)	leshort		<10
 >>>>(8.s*16) string 32STUB for MS-DOS, 32rtm DOS extender
 >>>>(8.s*16) string !32STUB for MS Windows
 >>>>>(0x3c.l+22)	leshort&0x2000	>0	(DLL)
+!:mime	application/x-dosexec
 >>>>>(0x3c.l+92)	leshort		0	(unknown subsystem)
 >>>>>(0x3c.l+92)	leshort		1	(native)
+!:mime	application/x-dosexec
 >>>>>(0x3c.l+92)	leshort		2	(GUI)
+!:mime	application/x-dosexec
 >>>>>(0x3c.l+92)	leshort		3	(console)
+!:mime	application/x-dosexec
 >>>>>(0x3c.l+92)	leshort		7	(POSIX)
 >>>(0x3c.l+92)	leshort		10	(EFI application)
 >>>(0x3c.l+92)	leshort		11	(EFI boot service driver)
@@ -97,6 +121,7 @@
 >>>(0x3c.l+92)	leshort		13	(XBOX)
 >>>(0x3c.l+4)	leshort		0x0	unknown processor
 >>>(0x3c.l+4)	leshort		0x14c	Intel 80386
+!:mime	application/x-dosexec
 >>>(0x3c.l+4)	leshort		0x166	MIPS R4000
 >>>(0x3c.l+4)	leshort		0x184	Alpha
 >>>(0x3c.l+4)	leshort		0x268	Motorola 68000
@@ -104,6 +129,7 @@
 >>>(0x3c.l+4)	leshort		0x290	PA-RISC
 >>>(0x3c.l+4)	leshort		0x200	Intel Itanium
 >>>(0x3c.l+22)	leshort&0x0100	>0	32-bit
+!:mime	application/x-dosexec
 >>>(0x3c.l+22)	leshort&0x1000	>0	system file
 >>>(0x3c.l+232) lelong	>0	Mono/.Net assembly
 
@@ -135,6 +161,7 @@
 >>>>0x30		string		Inno \b, InnoSetup self-extracting archive
 
 >>(0x3c.l) string !PE\0\0 MS-DOS executable
+!:mime	application/x-dosexec
 
 >>(0x3c.l)		string		NE \b, NE
 >>>(0x3c.l+0x36)	byte		0 (unknown OS)
@@ -169,6 +196,7 @@
 >>(0x3c.l)		string		W3 \b, W3 for MS Windows
 
 >>(0x3c.l)		string		LE\0\0 \b, LE executable
+!:mime	application/x-dosexec
 >>>(0x3c.l+0x0a)	leshort		1
 # some DOS extenders use LE files with OS/2 header
 >>>>0x240		search/0x100	DOS/4G for MS-DOS, DOS4GW DOS extender
@@ -186,8 +214,11 @@
 #>>>>(0x3c.l+0x1c)	lelong		>0x10000 for OS/2
 # fails with DOS-Extenders.
 >>>(0x3c.l+0x0a)	leshort		2 for MS Windows
+!:mime	application/x-dosexec
 >>>(0x3c.l+0x0a)	leshort		3 for DOS
+!:mime	application/x-dosexec
 >>>(0x3c.l+0x0a)	leshort		4 for MS Windows (VxD)
+!:mime	application/x-dosexec
 >>>(&0x7c.l+0x26)	string		UPX \b, UPX compressed
 >>>&(&0x54.l-3)		string		UNACE \b, ACE self-extracting archive
 
@@ -203,6 +234,7 @@
 >>>>&(2.s-514)	string	!LE
 >>>>>&-2	string	!BW \b, MZ for MS-DOS
 >>>>&(2.s-514)	string	LE \b, LE
+!:mime	application/x-dosexec
 >>>>>0x240	search/0x100	DOS/4G for MS-DOS, DOS4GW DOS extender
 # educated guess since indirection is still not capable enough for complex offset
 # calculations (next embedded executable would be at &(&2*512+&0-2)
@@ -255,7 +287,9 @@
 
 # skip to the end of the exe
 >(4.s*512)	long	x 
+!:mime	application/x-dosexec
 >>&(2.s-517)	byte	x 
+!:mime	application/x-dosexec
 >>>&0	string		PK\3\4 \b, ZIP self-extracting archive
 >>>&0	string		Rar! \b, RAR self-extracting archive
 >>>&0	string		=!\x11 \b, AIN 2.x self-extracting archive





--------
This message has been scanned by ComplianceSafe, powered by Palisade's PacketSure.


From christos at zoulas.com  Tue Oct 26 01:49:28 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 25 Oct 2010 18:49:28 -0400
Subject: Incorrect mime info for windows executables
In-Reply-To: <0784D725-E328-4942-A092-BE5C549723FD@palisadesys.com>
	from Guy Helmer (Oct 25,  4:17pm)
Message-ID: <20101025224928.4FD4556426@rebar.astron.com>

On Oct 25,  4:17pm, ghelmer at palisadesys.com (Guy Helmer) wrote:
-- Subject: Re: Incorrect mime info for windows executables

| On Oct 22, 2010, at 9:14 AM, Julien Pinon wrote:
| 
| > Hi,
| > 
| > I allways  get application/octet-stream  mime type for  some dos/windows
| > executables with any 5-0x release. For instance with putty.exe
| > (http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe) :
| > 
| > with libmagic 4-26:
| > 
| > vz-3-12415:/tmp# file putty.exe
| > putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
| > 
| > vz-3-12415:/tmp# file -i putty.exe
| > putty.exe: application/x-dosexec
| > 
| > 
| > with libmagic 5-04:
| > 
| > vz-2-12415:/tmp# file putty.exe
| > putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
| > 
| > vz-2-12415:/tmp# file -i putty.exe
| > putty.exe: application/octet-stream; charset=binary
| > 
| > 
| > I was  not able to  find if this  is a known problem  or not. Do  I miss
| > something ?
| > 
| 
| 
| This is the change I am using (against the magic files from 5.03) for DOS/Windows executables.
| 

The patch does not apply cleanly because this file has been changed. Can
you try to apply your patch on the following?

Thanks,

christos

#------------------------------------------------------------------------------
# $File: msdos,v 1.69 2010/09/20 18:55:20 rrt Exp $
# msdos:  file(1) magic for MS-DOS files
#

# .BAT files (Daniel Quinlan, quinlan at yggdrasil.com)
# updated by Joerg Jenderek at Oct 2008
0	string	@			
>1	string/cW	\ echo\ off	DOS batch file text
!:mime	text/x-msdos-batch
>1	string/cW	echo\ off	DOS batch file text
!:mime	text/x-msdos-batch
>1	string/cW	rem\ 		DOS batch file text
!:mime	text/x-msdos-batch
>1	string/cW	set\ 		DOS batch file text
!:mime	text/x-msdos-batch


# OS/2 batch files are REXX. the second regex is a bit generic, oh well
# the matched commands seem to be common in REXX and uncommon elsewhere
100	search/0xffff   rxfuncadd
>100	regex/c =^[\ \t]{0,10}call[\ \t]{1,10}rxfunc	OS/2 REXX batch file text
100	search/0xffff   say
>100	regex/c =^[\ \t]{0,10}say\ ['"]			OS/2 REXX batch file text

0	leshort		0x14c	MS Windows COFF Intel 80386 object file
#>4	ledate		x	stamp %s
0	leshort		0x166	MS Windows COFF MIPS R4000 object file
#>4	ledate		x	stamp %s
0	leshort		0x184	MS Windows COFF Alpha object file
#>4	ledate		x	stamp %s
0	leshort		0x268	MS Windows COFF Motorola 68000 object file
#>4	ledate		x	stamp %s
0	leshort		0x1f0	MS Windows COFF PowerPC object file
#>4	ledate		x	stamp %s
0	leshort		0x290	MS Windows COFF PA-RISC object file
#>4	ledate		x	stamp %s

# Tests for various EXE types.
#
# Many of the compressed formats were extraced from IDARC 1.23 source code.
#
0	string	MZ
!:mime	application/x-dosexec
# All non-DOS EXE extensions have the relocation table more than 0x40 bytes into the file.
>0x18	leshort <0x40 MS-DOS executable
# These traditional tests usually work but not always.  When test quality support is
# implemented these can be turned on.
#>>0x18	leshort	0x1c	(Borland compiler)
#>>0x18	leshort	0x1e	(MS compiler)

# If the relocation table is 0x40 or more bytes into the file, it's definitely
# not a DOS EXE.
>0x18  leshort >0x3f

# Maybe it's a PE?
>>(0x3c.l) string PE\0\0 PE
>>>(0x3c.l+24)	leshort		0x010b	\b32 executable
>>>(0x3c.l+24)	leshort		0x020b	\b32+ executable
>>>(0x3c.l+24)	leshort		0x0107	ROM image
>>>(0x3c.l+24)	default		x	Unknown PE signature
>>>>&0 		leshort		x	0x%x
>>>(0x3c.l+22)	leshort&0x2000	>0	(DLL)
>>>(0x3c.l+92)	leshort		1	(native)
>>>(0x3c.l+92)	leshort		2	(GUI)
>>>(0x3c.l+92)	leshort		3	(console)
>>>(0x3c.l+92)	leshort		7	(POSIX)
>>>(0x3c.l+92)	leshort		9	(Windows CE)
>>>(0x3c.l+92)	leshort		10	(EFI application)
>>>(0x3c.l+92)	leshort		11	(EFI boot service driver)
>>>(0x3c.l+92)	leshort		12	(EFI runtime driver)
>>>(0x3c.l+92)	leshort		13	(EFI ROM)
>>>(0x3c.l+92)	leshort		14	(XBOX)
>>>(0x3c.l+92)	leshort		15	(Windows boot application)
>>>(0x3c.l+92)	default		x	(Unknown subsystem
>>>>&0		leshort		x	0x%x)
>>>(0x3c.l+4)	leshort		0x14c	Intel 80386
>>>(0x3c.l+4)	leshort		0x166	MIPS R4000
>>>(0x3c.l+4)	leshort		0x168	MIPS R10000
>>>(0x3c.l+4)	leshort		0x184	Alpha
>>>(0x3c.l+4)	leshort		0x1a2	Hitachi SH3
>>>(0x3c.l+4)	leshort		0x1a6	Hitachi SH4
>>>(0x3c.l+4)	leshort		0x1c0	ARM
>>>(0x3c.l+4)	leshort		0x1c2	ARM Thumb
>>>(0x3c.l+4)	leshort		0x1f0	PowerPC
>>>(0x3c.l+4)	leshort		0x200	Intel Itanium
>>>(0x3c.l+4)	leshort		0x266	MIPS16
>>>(0x3c.l+4)	leshort		0x268	Motorola 68000
>>>(0x3c.l+4)	leshort		0x290	PA-RISC
>>>(0x3c.l+4)	leshort		0x366	MIPSIV
>>>(0x3c.l+4)	leshort		0x466	MIPS16 with FPU
>>>(0x3c.l+4)	leshort		0xebc	EFI byte code
>>>(0x3c.l+4)	leshort		0x8664	x86-64
>>>(0x3c.l+4)	leshort		0xc0ee	MSIL
>>>(0x3c.l+4)	default		x	Unknown processor type
>>>>&0		leshort		x	0x%x
>>>(0x3c.l+22)	leshort&0x0200	>0	(stripped to external PDB)
>>>(0x3c.l+22)	leshort&0x1000	>0	system file
>>>(0x3c.l+24)	leshort		0x010b
>>>>(0x3c.l+232) lelong	>0	Mono/.Net assembly
>>>(0x3c.l+24)	leshort		0x020b
>>>>(0x3c.l+248) lelong	>0	Mono/.Net assembly

# hooray, there's a DOS extender using the PE format, with a valid PE
# executable inside (which just prints a message and exits if run in win)
>>>(8.s*16)		string		32STUB	\b, 32rtm DOS extender
>>>(8.s*16)		string		!32STUB	\b, for MS Windows
>>>(0x3c.l+0xf8)	string		UPX0 \b, UPX compressed
>>>(0x3c.l+0xf8)	search/0x140	PEC2 \b, PECompact2 compressed
>>>(0x3c.l+0xf8)	search/0x140	UPX2
>>>>(&0x10.l+(-4))	string		PK\3\4 \b, ZIP self-extracting archive (Info-Zip)
>>>(0x3c.l+0xf8)	search/0x140	.idata
>>>>(&0xe.l+(-4))	string		PK\3\4 \b, ZIP self-extracting archive (Info-Zip)
>>>>(&0xe.l+(-4))	string		ZZ0 \b, ZZip self-extracting archive
>>>>(&0xe.l+(-4))	string		ZZ1 \b, ZZip self-extracting archive
>>>(0x3c.l+0xf8)	search/0x140	.rsrc
>>>>(&0x0f.l+(-4))	string		a\\\4\5 \b, WinHKI self-extracting archive
>>>>(&0x0f.l+(-4))	string		Rar! \b, RAR self-extracting archive
>>>>(&0x0f.l+(-4))	search/0x3000	MSCF \b, InstallShield self-extracting archive
>>>>(&0x0f.l+(-4))	search/32	Nullsoft \b, Nullsoft Installer self-extracting archive
>>>(0x3c.l+0xf8)	search/0x140	.data
>>>>(&0x0f.l)		string		WEXTRACT \b, MS CAB-Installer self-extracting archive
>>>(0x3c.l+0xf8)	search/0x140	.petite\0 \b, Petite compressed
>>>>(0x3c.l+0xf7)	byte		x
>>>>>(&0x104.l+(-4))	string		=!sfx! \b, ACE self-extracting archive
>>>(0x3c.l+0xf8)	search/0x140	.WISE \b, WISE installer self-extracting archive
>>>(0x3c.l+0xf8)	search/0x140	.dz\0\0\0 \b, Dzip self-extracting archive
>>>&(0x3c.l+0xf8)	search/0x100	_winzip_ \b, ZIP self-extracting archive (WinZip)
>>>&(0x3c.l+0xf8)	search/0x100	SharedD \b, Microsoft Installer self-extracting archive
>>>0x30			string		Inno \b, InnoSetup self-extracting archive

# Hmm, not a PE but the relocation table is too high for a traditional DOS exe,
# must be one of the unusual subformats.
>>(0x3c.l) string !PE\0\0 MS-DOS executable

>>(0x3c.l)		string		NE \b, NE
>>>(0x3c.l+0x36)	byte		1 for OS/2 1.x
>>>(0x3c.l+0x36)	byte		2 for MS Windows 3.x
>>>(0x3c.l+0x36)	byte		3 for MS-DOS
>>>(0x3c.l+0x36)	byte		4 for Windows 386
>>>(0x3c.l+0x36)	byte		5 for Borland Operating System Services
>>>(0x3c.l+0x36)	default		x
>>>>(0x3c.l+0x36)	byte		x (unknown OS %x)
>>>(0x3c.l+0x36)	byte		0x81 for MS-DOS, Phar Lap DOS extender
>>>(0x3c.l+0x0c)	leshort&0x8003	0x8002 (DLL)
>>>(0x3c.l+0x0c)	leshort&0x8003	0x8001 (driver)
>>>&(&0x24.s-1)		string		ARJSFX \b, ARJ self-extracting archive
>>>(0x3c.l+0x70)	search/0x80	WinZip(R)\ Self-Extractor \b, ZIP self-extracting archive (WinZip)

>>(0x3c.l)		string		LX\0\0 \b, LX
>>>(0x3c.l+0x0a)	leshort		<1 (unknown OS)
>>>(0x3c.l+0x0a)	leshort		1 for OS/2
>>>(0x3c.l+0x0a)	leshort		2 for MS Windows
>>>(0x3c.l+0x0a)	leshort		3 for DOS
>>>(0x3c.l+0x0a)	leshort		>3 (unknown OS)
>>>(0x3c.l+0x10)	lelong&0x28000	=0x8000 (DLL)
>>>(0x3c.l+0x10)	lelong&0x20000	>0 (device driver)
>>>(0x3c.l+0x10)	lelong&0x300	0x300 (GUI)
>>>(0x3c.l+0x10)	lelong&0x28300	<0x300 (console)
>>>(0x3c.l+0x08)	leshort		1 i80286
>>>(0x3c.l+0x08)	leshort		2 i80386
>>>(0x3c.l+0x08)	leshort		3 i80486
>>>(8.s*16)		string		emx \b, emx
>>>>&1			string		x %s
>>>&(&0x54.l-3)		string		arjsfx \b, ARJ self-extracting archive

# MS Windows system file, supposedly a collection of LE executables
>>(0x3c.l)		string		W3 \b, W3 for MS Windows

>>(0x3c.l)		string		LE\0\0 \b, LE executable
>>>(0x3c.l+0x0a)	leshort		1
# some DOS extenders use LE files with OS/2 header
>>>>0x240		search/0x100	DOS/4G for MS-DOS, DOS4GW DOS extender
>>>>0x240		search/0x200	WATCOM\ C/C++ for MS-DOS, DOS4GW DOS extender
>>>>0x440		search/0x100	CauseWay\ DOS\ Extender for MS-DOS, CauseWay DOS extender
>>>>0x40		search/0x40	PMODE/W for MS-DOS, PMODE/W DOS extender
>>>>0x40		search/0x40	STUB/32A for MS-DOS, DOS/32A DOS extender (stub)
>>>>0x40		search/0x80	STUB/32C for MS-DOS, DOS/32A DOS extender (configurable stub)
>>>>0x40		search/0x80	DOS/32A for MS-DOS, DOS/32A DOS extender (embedded)
# this is a wild guess; hopefully it is a specific signature
>>>>&0x24		lelong		<0x50
>>>>>(&0x4c.l)		string		\xfc\xb8WATCOM
>>>>>>&0		search/8	3\xdbf\xb9 \b, 32Lite compressed
# another wild guess: if real OS/2 LE executables exist, they probably have higher start EIP
#>>>>(0x3c.l+0x1c)	lelong		>0x10000 for OS/2
# fails with DOS-Extenders.
>>>(0x3c.l+0x0a)	leshort		2 for MS Windows
>>>(0x3c.l+0x0a)	leshort		3 for DOS
>>>(0x3c.l+0x0a)	leshort		4 for MS Windows (VxD)
>>>(&0x7c.l+0x26)	string		UPX \b, UPX compressed
>>>&(&0x54.l-3)		string		UNACE \b, ACE self-extracting archive

# looks like ASCII, probably some embedded copyright message.
# and definitely not NE/LE/LX/PE
>>0x3c		lelong	>0x20000000
>>>(4.s*512)	leshort !0x014c \b, MZ for MS-DOS
# header data too small for extended executable
>2		long	!0
>>0x18		leshort <0x40
>>>(4.s*512)	leshort !0x014c

>>>>&(2.s-514)	string	!LE
>>>>>&-2	string	!BW \b, MZ for MS-DOS
>>>>&(2.s-514)	string	LE \b, LE
>>>>>0x240	search/0x100	DOS/4G for MS-DOS, DOS4GW DOS extender
# educated guess since indirection is still not capable enough for complex offset
# calculations (next embedded executable would be at &(&2*512+&0-2)
# I suspect there are only LE executables in these multi-exe files
>>>>&(2.s-514)	string	BW
>>>>>0x240	search/0x100	DOS/4G ,\b LE for MS-DOS, DOS4GW DOS extender (embedded)
>>>>>0x240	search/0x100	!DOS/4G ,\b BW collection for MS-DOS

# This sequence skips to the first COFF segment, usually .text
>(4.s*512)	leshort		0x014c \b, COFF
>>(8.s*16)	string		go32stub for MS-DOS, DJGPP go32 DOS extender
>>(8.s*16)	string		emx
>>>&1		string		x for DOS, Win or OS/2, emx %s
>>&(&0x42.l-3)	byte		x 
>>>&0x26	string		UPX \b, UPX compressed
# and yet another guess: small .text, and after large .data is unusal, could be 32lite
>>&0x2c		search/0xa0	.text
>>>&0x0b	lelong		<0x2000
>>>>&0		lelong		>0x6000 \b, 32lite compressed

>(8.s*16) string $WdX \b, WDos/X DOS extender

# By now an executable type should have been printed out.  The executable
# may be a self-uncompressing archive, so look for evidence of that and 
# print it out.  
#
# Some signatures below from Greg Roelofs, newt at uchicago.edu.
#
>0x35	string	\x8e\xc0\xb9\x08\x00\xf3\xa5\x4a\x75\xeb\x8e\xc3\x8e\xd8\x33\xff\xbe\x30\x00\x05 \b, aPack compressed
>0xe7	string	LH/2\ 	Self-Extract \b, %s
>0x1c	string	UC2X	\b, UCEXE compressed
>0x1c	string	WWP\ 	\b, WWPACK compressed
>0x1c	string	RJSX 	\b, ARJ self-extracting archive
>0x1c	string	diet 	\b, diet compressed
>0x1c	string	LZ09 	\b, LZEXE v0.90 compressed
>0x1c	string	LZ91 	\b, LZEXE v0.91 compressed
>0x1c	string	tz 	\b, TinyProg compressed
>0x1e	string	Copyright\ 1989-1990\ PKWARE\ Inc.	Self-extracting PKZIP archive
!:mime	application/zip
# Yes, this really is "Copr", not "Corp."
>0x1e	string	PKLITE\ Copr.	Self-extracting PKZIP archive
!:mime	application/zip
# winarj stores a message in the stub instead of the sig in the MZ header
>0x20	search/0xe0	aRJsfX \b, ARJ self-extracting archive
>0x20	string AIN
>>0x23	string 2	\b, AIN 2.x compressed
>>0x23	string <2	\b, AIN 1.x compressed
>>0x23	string >2	\b, AIN 1.x compressed
>0x24	string	LHa's\ SFX \b, LHa self-extracting archive
!:mime	application/x-lha
>0x24	string	LHA's\ SFX \b, LHa self-extracting archive
!:mime	application/x-lha
>0x24	string	\ $ARX \b, ARX self-extracting archive
>0x24	string	\ $LHarc \b, LHarc self-extracting archive
>0x20	string	SFX\ by\ LARC \b, LARC self-extracting archive
>0x40	string aPKG \b, aPackage self-extracting archive
>0x64	string	W\ Collis\0\0 \b, Compack compressed
>0x7a	string		Windows\ self-extracting\ ZIP	\b, ZIP self-extracting archive
>>&0xf4 search/0x140 \x0\x40\x1\x0
>>>(&0.l+(4)) string MSCF \b, WinHKI CAB self-extracting archive
>1638	string	-lh5- \b, LHa self-extracting archive v2.13S
>0x17888 string Rar! \b, RAR self-extracting archive

# Skip to the end of the EXE.  This will usually work fine in the PE case
# because the MZ image is hardcoded into the toolchain and almost certainly
# won't match any of these signatures.
>(4.s*512)	long	x 
>>&(2.s-517)	byte	x 
>>>&0	string		PK\3\4 \b, ZIP self-extracting archive
>>>&0	string		Rar! \b, RAR self-extracting archive
>>>&0	string		=!\x11 \b, AIN 2.x self-extracting archive
>>>&0	string		=!\x12 \b, AIN 2.x self-extracting archive
>>>&0	string		=!\x17 \b, AIN 1.x self-extracting archive
>>>&0	string		=!\x18 \b, AIN 1.x self-extracting archive
>>>&7	search/400	**ACE** \b, ACE self-extracting archive
>>>&0	search/0x480	UC2SFX\ Header \b, UC2 self-extracting archive

# a few unknown ZIP sfxes, no idea if they are needed or if they are
# already captured by the generic patterns above
>(8.s*16)	search/0x20	PKSFX \b, ZIP self-extracting archive (PKZIP)
# TODO: how to add this? >FileSize-34 string Windows\ Self-Installing\ Executable \b, ZIP self-extracting archive
#

# TELVOX Teleinformatica CODEC self-extractor for OS/2:
>49801	string	\x79\xff\x80\xff\x76\xff	\b, CODEC archive v3.21
>>49824 leshort		=1			\b, 1 file
>>49824 leshort		>1			\b, %u files

# .COM formats (Daniel Quinlan, quinlan at yggdrasil.com)
# Uncommenting only the first two lines will cover about 2/3 of COM files,
# but it isn't feasible to match all COM files since there must be at least
# two dozen different one-byte "magics".
# test too generic ?
0	byte		0xe9		DOS executable (COM)
>0x1FE leshort		0xAA55		\b, boot code
>6	string		SFX\ of\ LHarc	(%s)
0	belong	0xffffffff		DOS executable (device driver)
#CMD640X2.SYS
>10	string	>\x23			
>>10	string	!\x2e			
>>>17	string	<\x5B			
>>>>10	string	x			\b, name: %.8s
#UDMA.SYS KEYB.SYS CMD640X2.SYS
>10	string	<\x41			
>>12	string	>\x40			
>>>10	string	!$			
>>>>12	string	x			\b, name: %.8s
#BTCDROM.SYS ASPICD.SYS
>22	string	>\x40			
>>22	string	<\x5B			
>>>23	string	<\x5B			
>>>>22	string	x			\b, name: %.8s
#ATAPICD.SYS
>76	string	\0			
>>77	string	>\x40			
>>>77	string	<\x5B			
>>>>77	string	x			\b, name: %.8s
# test too generic ?
0	byte		0x8c		DOS executable (COM)
# updated by Joerg Jenderek at Oct 2008
0	ulelong		0xffff10eb	DR-DOS executable (COM)
# byte 0xeb conflicts with "sequent" magic leshort 0xn2eb
0	ubeshort&0xeb8d	>0xeb00		
# DR-DOS STACKER.COM SCREATE.SYS missed
>0	byte		0xeb		DOS executable (COM)
>>0x1FE leshort		0xAA55		\b, boot code
>>85	string		UPX		\b, UPX compressed
>>4	string		\ $ARX		\b, ARX self-extracting archive
>>4	string		\ $LHarc	\b, LHarc self-extracting archive
>>0x20e string		SFX\ by\ LARC	\b, LARC self-extracting archive
# updated by Joerg Jenderek at Oct 2008
#0	byte		0xb8		COM executable
0	uleshort&0x80ff	0x00b8		
# modified by Joerg Jenderek
>1	lelong		!0x21cd4cff	COM executable for DOS
# http://syslinux.zytor.com/comboot.php
# (32-bit COMBOOT) programs *.C32 contain 32-bit code and run in flat-memory 32-bit protected mode
# start with assembler instructions mov eax,21cd4cffh
0	uleshort&0xc0ff	0xc0b8		
>1	lelong		0x21cd4cff	COM executable (32-bit COMBOOT)
# start with assembler instructions mov eax,21cd4cfeh
0	uleshort&0xc0ff	0xc0b8		
>1	lelong		0x21cd4cfe	COM executable (32-bit COMBOOT, relocatable)
0	string	\x81\xfc		
>4	string	\x77\x02\xcd\x20\xb9	
>>36	string	UPX!			FREE-DOS executable (COM), UPX compressed
252	string Must\ have\ DOS\ version DR-DOS executable (COM)
# added by Joerg Jenderek at Oct 2008
# GRR search is not working
#34	search/2	UPX!		FREE-DOS executable (COM), UPX compressed
34	string	UPX!			FREE-DOS executable (COM), UPX compressed
35	string	UPX!			FREE-DOS executable (COM), UPX compressed
# GRR search is not working
#2	search/28	\xcd\x21	COM executable for MS-DOS
#WHICHFAT.cOM
2	string	\xcd\x21		COM executable for DOS
#DELTREE.cOM DELTREE2.cOM
4	string	\xcd\x21		COM executable for DOS
#IFMEMDSK.cOM ASSIGN.cOM COMP.cOM
5	string	\xcd\x21		COM executable for DOS
#DELTMP.COm HASFAT32.cOM
7	string	\xcd\x21		
>0	byte	!0xb8			COM executable for DOS
#COMP.cOM MORE.COm
10	string	\xcd\x21		
>5	string	!\xcd\x21		COM executable for DOS
#comecho.com
13	string	\xcd\x21		COM executable for DOS
#HELP.COm EDIT.coM
18	string	\xcd\x21		COM executable for MS-DOS
#NWRPLTRM.COm
23	string	\xcd\x21		COM executable for MS-DOS
#LOADFIX.cOm LOADFIX.cOm
30	string	\xcd\x21		COM executable for MS-DOS
#syslinux.com 3.11
70	string	\xcd\x21		COM executable for DOS
# many compressed/converted COMs start with a copy loop instead of a jump
0x6	search/0xa	\xfc\x57\xf3\xa5\xc3	COM executable for MS-DOS
0x6	search/0xa	\xfc\x57\xf3\xa4\xc3	COM executable for DOS
>0x18	search/0x10	\x50\xa4\xff\xd5\x73	\b, aPack compressed
0x3c	string		W\ Collis\0\0		COM executable for MS-DOS, Compack compressed
# FIXME: missing diet .com compression

# miscellaneous formats
0	string		LZ		MS-DOS executable (built-in)
#0	byte		0xf0		MS-DOS program library data
#

# AAF files:
# <stuartc at rd.bbc.co.uk> Stuart Cunningham
0	string	\320\317\021\340\241\261\032\341AAFB\015\000OM\006\016\053\064\001\001\001\377			AAF legacy file using MS Structured Storage
>30	byte	9		(512B sectors)
>30	byte	12		(4kB sectors)
0	string	\320\317\021\340\241\261\032\341\001\002\001\015\000\002\000\000\006\016\053\064\003\002\001\001			AAF file using MS Structured Storage
>30	byte	9		(512B sectors)
>30	byte	12		(4kB sectors)

# Popular applications
2080	string	Microsoft\ Word\ 6.0\ Document	%s
!:mime	application/msword
2080	string	Documento\ Microsoft\ Word\ 6 Spanish Microsoft Word 6 document data
!:mime	application/msword
# Pawel Wiecek <coven at i17linuxb.ists.pwr.wroc.pl> (for polish Word)
2112	string	MSWordDoc			Microsoft Word document data
!:mime	application/msword
#
0	belong	0x31be0000			Microsoft Word Document
!:mime	application/msword
#
0	string	PO^Q`				Microsoft Word 6.0 Document
!:mime	application/msword
#
0	string	\376\067\0\043			Microsoft Office Document
!:mime	application/msword
0	string	\333\245-\0\0\0			Microsoft Office Document
!:mime	application/msword
512	string		\354\245\301		Microsoft Word Document
!:mime	application/msword
#
2080	string	Microsoft\ Excel\ 5.0\ Worksheet	%s
!:mime	application/vnd.ms-excel

2080	string	Foglio\ di\ lavoro\ Microsoft\ Exce	%s
!:mime	application/vnd.ms-excel
#
# Pawel Wiecek <coven at i17linuxb.ists.pwr.wroc.pl> (for polish Excel)
2114	string	Biff5		Microsoft Excel 5.0 Worksheet
!:mime	application/vnd.ms-excel
# Italian MS-Excel
2121	string	Biff5		Microsoft Excel 5.0 Worksheet
!:mime	application/vnd.ms-excel
0	string	\x09\x04\x06\x00\x00\x00\x10\x00	Microsoft Excel Worksheet
!:mime	application/vnd.ms-excel
#
0	belong	0x00001a00	Lotus 1-2-3
!:mime	application/x-123
>4	belong	0x00100400	wk3 document data
>4	belong	0x02100400	wk4 document data
>4	belong	0x07800100	fm3 or fmb document data
>4	belong	0x07800000	fm3 or fmb document data
#
0	belong	0x00000200	Lotus 1-2-3
!:mime	application/x-123
>4	belong	0x06040600	wk1 document data
>4	belong	0x06800200	fmt document data
0	string		WordPro\0	Lotus WordPro
!:mime	application/vnd.lotus-wordpro
0	string		WordPro\r\373	Lotus WordPro
!:mime	application/vnd.lotus-wordpro


# Summary: Script used by InstallScield to uninstall applications
# Extension: .isu
# Submitted by: unknown
# Modified by (1): Abel Cheung <abelcheung at gmail.com> (replace useless entry)
0		string		\x71\xa8\x00\x00\x01\x02
>12		string		Stirling\ Technologies,		InstallShield Uninstall Script

# Winamp .avs
#0	string	Nullsoft\ AVS\ Preset\ \060\056\061\032 A plug in for Winamp ms-windows Freeware media player
0	string	Nullsoft\ AVS\ Preset\ 	Winamp plug in

# Windows Metafont .WMF
0	string	\327\315\306\232	ms-windows metafont .wmf
0	string	\002\000\011\000	ms-windows metafont .wmf
0	string	\001\000\011\000	ms-windows metafont .wmf

#tz3 files whatever that is (MS Works files)
0	string	\003\001\001\004\070\001\000\000	tz3 ms-works file
0	string	\003\002\001\004\070\001\000\000	tz3 ms-works file
0	string	\003\003\001\004\070\001\000\000	tz3 ms-works file

# PGP sig files .sig
#0 string \211\000\077\003\005\000\063\237\127 065 to  \027\266\151\064\005\045\101\233\021\002 PGP sig
0 string \211\000\077\003\005\000\063\237\127\065\027\266\151\064\005\045\101\233\021\002 PGP sig
0 string \211\000\077\003\005\000\063\237\127\066\027\266\151\064\005\045\101\233\021\002 PGP sig
0 string \211\000\077\003\005\000\063\237\127\067\027\266\151\064\005\045\101\233\021\002 PGP sig
0 string \211\000\077\003\005\000\063\237\127\070\027\266\151\064\005\045\101\233\021\002 PGP sig
0 string \211\000\077\003\005\000\063\237\127\071\027\266\151\064\005\045\101\233\021\002 PGP sig
0 string \211\000\225\003\005\000\062\122\207\304\100\345\042 PGP sig

# windows zips files .dmf
0	string	MDIF\032\000\010\000\000\000\372\046\100\175\001\000\001\036\001\000 MS Windows special zipped file


#ico files
0	string	\102\101\050\000\000\000\056\000\000\000\000\000\000\000	Icon for MS Windows

# Windows icons (Ian Springer <ips at fpk.hp.com>)
0	string	\000\000\001\000	MS Windows icon resource
!:mime	image/x-icon
>4	byte	1			- 1 icon
>4	byte	>1			- %d icons
>>6	byte	>0			\b, %dx
>>>7	byte	>0			\b%d
>>8	byte	0			\b, 256-colors
>>8	byte	>0			\b, %d-colors


# .chr files
0	string	PK\010\010BGI	Borland font 
>4	string	>\0	%s
# then there is a copyright notice


# .bgi files
0	string	pk\010\010BGI	Borland device 
>4	string	>\0	%s
# then there is a copyright notice


# Windows Recycle Bin record file (named INFO2)
# By Abel Cheung (abelcheung AT gmail dot com)
# Version 4 always has 280 bytes (0x118) per record, version 5 has 800 bytes
# Since Vista uses another structure, INFO2 structure probably won't change
# anymore. Detailed analysis in:
# http://www.cybersecurityinstitute.biz/downloads/INFO2.pdf
0	lelong		0x00000004
>12	lelong		0x00000118	Windows Recycle Bin INFO2 file (Win98 or below)

0	lelong		0x00000005
>12	lelong		0x00000320	Windows Recycle Bin INFO2 file (Win2k - WinXP)


##### put in Either Magic/font or Magic/news
# Acroread or something	 files wrongly identified as G3	 .pfm
# these have the form \000 \001 any? \002 \000 \000
# or \000 \001 any? \022 \000 \000
0	belong&0xffff00ff	0x00010012	PFM data
>4	string			\000\000
>6	string			>\060		- %s

0	belong&0xffff00ff	0x00010002	PFM data
>4	string			\000\000
>6	string			>\060		- %s
#0	string	\000\001 pfm?
#>3	string	\022\000\000Copyright\	yes
#>3	string	\002\000\000Copyright\	yes
#>3	string	>\0	oops, not a font file. Cancel that.
#it clashes with ttf files so put it lower down.

# From Doug Lee via a FreeBSD pr
9	string		GERBILDOC	First Choice document
9	string		GERBILDB	First Choice database
9	string		GERBILCLIP	First Choice database
0	string		GERBIL		First Choice device file
9	string		RABBITGRAPH	RabbitGraph file
0	string		DCU1		Borland Delphi .DCU file
0	string		=!<spell>	MKS Spell hash list (old format)
0	string		=!<spell2>	MKS Spell hash list
# Too simple - MPi
#0	string		AH		Halo(TM) bitmapped font file
0	lelong		0x08086b70	TurboC BGI file
0	lelong		0x08084b50	TurboC Font file

# WARNING: below line conflicts with Infocom game data Z-machine 3
0	byte		0x03		DBase 3 data file
>0x04	lelong		0		(no records)
>0x04	lelong		>0		(%ld records)
0	byte		0x83		DBase 3 data file with memo(s)
>0x04	lelong		0		(no records)
>0x04	lelong		>0		(%ld records)
0	leshort		0x0006		DBase 3 index file
0	string		PMCC		Windows 3.x .GRP file
1	string		RDC-meg		MegaDots 
>8	byte		>0x2F		version %c
>9	byte		>0x2F		\b.%c file
0	lelong		0x4C
>4	lelong		0x00021401	Windows shortcut file

# DOS EPS Binary File Header
# From: Ed Sznyter <ews at Black.Market.NET>
0	belong		0xC5D0D3C6	DOS EPS Binary File
>4	long		>0		Postscript starts at byte %d
>>8	long		>0		length %d
>>>12	long		>0		Metafile starts at byte %d
>>>>16	long		>0		length %d
>>>20	long		>0		TIFF starts at byte %d
>>>>24	long		>0		length %d

# TNEF magic From "Joomy" <joomy at se-ed.net> 
# Microsoft Outlook's Transport Neutral Encapsulation Format (TNEF)
0	leshort		0x223e9f78	TNEF
!:mime	application/vnd.ms-tnef

# HtmlHelp files (.chm)
0	string	ITSF\003\000\000\000\x60\000\000\000\001\000\000\000	MS Windows HtmlHelp Data

# GFA-BASIC (Wolfram Kleff)
2	string		GFA-BASIC3	GFA-BASIC 3 data

#------------------------------------------------------------------------------
# From Stuart Caie <kyzer at 4u.net> (developer of cabextract)
# Microsoft Cabinet files
0	string		MSCF\0\0\0\0	Microsoft Cabinet archive data
!:mime application/vnd.ms-cab-compressed
>8	lelong		x		\b, %u bytes
>28	leshort		1		\b, 1 file
>28	leshort		>1		\b, %u files

# InstallShield Cabinet files
0	string		ISc(		InstallShield Cabinet archive data
>5	byte&0xf0	=0x60		version 6,
>5	byte&0xf0	!0x60		version 4/5,
>(12.l+40)	lelong	x		%u files

# Windows CE package files
0	string		MSCE\0\0\0\0	Microsoft WinCE install header
>20	lelong		0		\b, architecture-independent
>20	lelong		103		\b, Hitachi SH3
>20	lelong		104		\b, Hitachi SH4
>20	lelong		0xA11		\b, StrongARM
>20	lelong		4000		\b, MIPS R4000
>20	lelong		10003		\b, Hitachi SH3
>20	lelong		10004		\b, Hitachi SH3E
>20	lelong		10005		\b, Hitachi SH4
>20	lelong		70001		\b, ARM 7TDMI
>52	leshort		1		\b, 1 file
>52	leshort		>1		\b, %u files
>56	leshort		1		\b, 1 registry entry
>56	leshort		>1		\b, %u registry entries


# Windows Enhanced Metafile (EMF)
# See msdn.microsoft.com/archive/en-us/dnargdi/html/msdn_enhmeta.asp 
# for further information.
0	ulelong 1
>40	string	\ EMF		Windows Enhanced Metafile (EMF) image data
>>44	ulelong x		version 0x%x

# From: Alex Beregszaszi <alex at fsn.hu>
0	string	COWD		VMWare3
>4	byte	3		disk image
>>32	lelong	x		(%d/
>>36	lelong	x		\b%d/
>>40	lelong	x		\b%d)
>4	byte	2		undoable disk image
>>32	string	>\0		(%s)

0	string	VMDK		 VMware4 disk image
0	string	KDMV		 VMware4 disk image

#--------------------------------------------------------------------
# Qemu Emulator Images
# Lines written by Friedrich Schwittay (f.schwittay at yousable.de)
# Updated by Adam Buchbinder (adam.buchbinder at gmail.com)
# Made by reading sources, reading documentation, and doing trial and error
# on existing QCOW files
0	string	QFI\xFB	QEMU QCOW Image

# Uncomment the following line to display Magic (only used for debugging
# this magic number)
#>0	string	x	, Magic: %s

# There are currently 2 Versions: "1" and "2".
# http://www.gnome.org/~markmc/qcow-image-format-version-1.html
>4	belong	1	(v1)

# Using the existence of the Backing File Offset to determine whether
# to read Backing File Information
>>12	belong	 >0	 \b, has backing file (
# Note that this isn't a null-terminated string; the length is actually
# (16.L). Assuming a null-terminated string happens to work usually, but it
# may spew junk until it reaches a \0 in some cases.
>>>(12.L)	 string >\0	\bpath %s

# Modification time of the Backing File
# Really useful if you want to know if your backing
# file is still usable together with this image
>>>>20	bedate >0	\b, mtime %s)
>>>>20	default x	\b)

# Size is stored in bytes in a big-endian u64.
>>24	bequad	x	 \b, %lld bytes

# 1 for AES encryption, 0 for none.
>>36	belong	1	\b, AES-encrypted

# http://www.gnome.org/~markmc/qcow-image-format.html
>4	belong	2	(v2)
# Using the existence of the Backing File Offset to determine whether
# to read Backing File Information
>>8	bequad  >0	 \b, has backing file
# Note that this isn't a null-terminated string; the length is actually
# (16.L). Assuming a null-terminated string happens to work usually, but it
# may spew junk until it reaches a \0 in some cases. Also, since there's no
# .Q modifier, we just use the bottom four bytes as an offset. Note that if
# the file is over 4G, and the backing file path is stored after the first 4G,
# the wrong filename will be printed. (This should be (8.Q), when that syntax
# is introduced.)
>>>(12.L)	 string >\0	(path %s)
>>24	bequad	x	\b, %lld bytes
>>32	belong	1	\b, AES-encrypted

>4	default x	(unknown version)

0	string	QEVM		QEMU suspend to disk image

0	string	Bochs\ Virtual\ HD\ Image	Bochs disk image,
>32	string	x				type %s,
>48	string	x				subtype %s

0	lelong	0x02468ace			Bochs Sparse disk image

# from http://filext.com by Derek M Jones <derek at knosof.co.uk>
# False positive with PPT (also currently this string is too long)
#0	string	\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x3E\x00\x03\x00\xFE\xFF\x09\x00\x06	Microsoft Installer
0	string	\320\317\021\340\241\261\032\341	Microsoft Office Document
#>48	byte	0x1B					Excel Document
#!:mime application/vnd.ms-excel
>546	string	bjbj			Microsoft Word Document
!:mime	application/msword
>546	string	jbjb			Microsoft Word Document
!:mime	application/msword

0	string	\224\246\056		Microsoft Word Document
!:mime	application/msword

512	string	R\0o\0o\0t\0\ \0E\0n\0t\0r\0y	Microsoft Word Document
!:mime	application/msword

# From: "Nelson A. de Oliveira" <naoliv at gmail.com>
# Magic type for Dell's BIOS .hdr files
# Dell's .hdr
0	string $RBU
>23	string Dell			%s system BIOS
>48	string x			version %.3s

# Type: Microsoft DirectDraw Surface
# URL:	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/directx9_c/directx/graphics/reference/DDSFileReference/ddsfileformat.asp
# From: Morten Hustveit <morten at debian.org>
0	string	DDS\040\174\000\000\000 Microsoft DirectDraw Surface (DDS),
>16	lelong	>0			%hd x
>12	lelong	>0			%hd,
>84	string	x			%.4s

# Type: Microsoft Document Imaging Format (.mdi)
# URL:	http://en.wikipedia.org/wiki/Microsoft_Document_Imaging_Format
# From: Daniele Sempione <scrows at oziosi.org>
0	short	0x5045			Microsoft Document Imaging Format

# MS eBook format (.lit)
0	string	ITOLITLS		Microsoft Reader eBook Data
>8	lelong	x			\b, version %u
!:mime					application/x-ms-reader

# Windows CE Binary Image Data Format
# From: Dr. Jesus <j at hug.gs>
0	string	B000FF\n	Windows Embedded CE binary image

# Windows Imaging (WIM) Image
0	string	MSWIM\000\000\000	Windows imaging (WIM) image


From cchittleborough+comp.file at cluemail.com  Tue Oct 26 09:42:08 2010
From: cchittleborough+comp.file at cluemail.com (Christopher Chittleborough)
Date: Tue, 26 Oct 2010 17:12:08 +1030
Subject: Patch to magdir/archive: better handling of ZIP-based formats
Message-ID: <1288075328.23702.1401962711@webmail.messagingengine.com>

Here's a patch to magic/magdir/archive (1.58 2010/09/20 19:19:16)
which improves the handling of ZIP and derived formats. It makes the
rules a little more robust, fixes some minor problems, removes some
duplication and adds explanatory comments. You can also get it from
http://cchittleborough.cluemail.com/archive.full.patch if necessary.

This patch also tidies up whitespace in some unrelated rules. If you
don't want this,
http://cchittleborough.cluemail.com/archive.minimal.patch
is an alternate version of the patch which only touches ZIP-related
rules.


--- file-CVS/magic/magdir/archive       2010-10-25 03:41:15.000000000
+0000
+++ file-NEW/magic/Magdir/archive       2010-10-26 06:08:11.000000000
+0000
@@ -276,7 +276,7 @@
 # Splint
 0       string  \x93\xb9\x06 Splint archive data
 # InstallShield
-0      string   \x13\x5d\x65\x8c InstallShield Z archive Data
+0      string  \x13\x5d\x65\x8c InstallShield Z archive Data
 # Gather
 1       string  GTH Gather archive data
 # BOA
@@ -497,7 +497,7 @@
 0       string          HPAK            HPACK archive data
 
 # JAM Archive volume format, by Dmitry.Kohmanyuk at UA.net
-0      string          \351,\001JAM\           JAM archive,
+0      string          \351,\001JAM\           JAM archive,
 >7	string		>\0			version %.4s
 >0x26	byte		=0x27			-
 >>0x2b	string          >\0			label %.11s,
@@ -569,13 +569,17 @@
 # Zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
 0       string          PK\003\004
 
-# Specialised zip formats
->30    string          mimetype
+# Specialised zip formats which start with a member named 'mimetype'
+# (stored uncompressed, with no 'extra field') containing the file's
MIME type.
+# Check for have 8-byte name, 0-byte extra field, name "mimetype", and
+#  contents starting with "application/":
+>26    string          \x8\0\0\0mimetypeapplication/
 
-# OpenOffice.org / KOffice / StarOffice documents
-# From: Abel Cheung <abel at oaka.org>
+#  KOffice / OpenOffice & StarOffice / OpenDocument formats
+#    From: Abel Cheung <abel at oaka.org>
 
-# KOffice (1.2 or above) formats
+#   KOffice (1.2 or above) formats
+#    (mimetype contains "application/vnd.kde.<SUBTYPE>")
 >>50	string	vnd.kde.		KOffice (>=1.2)
 >>>58	string	karbon			Karbon document
 >>>58	string	kchart			KChart document
@@ -586,8 +590,8 @@
 >>>58	string	kspread			KSpread document
 >>>58	string	kword			KWord document
 
-# OpenOffice formats (for OpenOffice 1.x / StarOffice 6/7)
-#  mimetype contains 'application/vnd.sun.xml.<SUBTYPE>'
+#   OpenOffice formats (for OpenOffice 1.x / StarOffice 6/7)
+#    (mimetype contains "application/vnd.sun.xml.<SUBTYPE>")
 >>50	string	vnd.sun.xml.		OpenOffice.org 1.x
 >>>62	string	writer			Writer
 >>>>68	byte	!0x2e			document
@@ -605,9 +609,9 @@
 >>>62	string	math			Math document
 >>>62	string	base			Database file
 
-# OpenDocument formats (for OpenOffice 2.x / StarOffice >= 8)
-# http://lists.oasis-open.org/archives/office/200505/msg00006.html
-#  mimetype contains 'application/vnd.oasis.opendocument.<SUBTYPE>'
+#   OpenDocument formats (for OpenOffice 2.x / StarOffice >= 8)
+#    http://lists.oasis-open.org/archives/office/200505/msg00006.html
+#    (mimetype contains "application/vnd.oasis.opendocument.<SUBTYPE>")
 >>50	string	vnd.oasis.opendocument.	OpenDocument
 >>>73	string	text
 >>>>77	byte	!0x2d			Text
@@ -651,48 +655,40 @@
 >>>>78	string	-template		Template
 !:mime  application/vnd.oasis.opendocument.image-template
 
-# OEBPS Container Format (OCF) 1.0
->>50   string  epub+zip                OEBPS Container Format (OCF) 1.0
(EPUB)
+#  EPUB (OEBPS) books using OCF (OEBPS Container Format)
+#    From: Adam Buchbinder <adam.buchbinder at gmail.com>
+#    http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm, section 4.
+#    (mimetype contains "application/epub+zip")
+>>50   string  epub+zip                EPUB ebook data
 !:mime  application/epub+zip
 
-# Generic zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
-# N.B. Must come after specialised formats
-# Next line simply matches anything, including "mimetype" as used above
->30    byte            x
->>4    byte            0x00            Zip archive data
-!:mime application/zip
->>4    byte            0x09            Zip archive data, at least v0.9
to extract
-!:mime application/zip
->>4    byte            0x0a            Zip archive data, at least v1.0
to extract
+#  Catch other ZIP-with-mimetype formats
+#      In a ZIP file, the bytes immediately after a member's contents
are
+#      always "PK". The 2 regex rules here print the "mimetype"
member's
+#      contents up to the first 'P'. Luckily, most MIME types don't
contain
+#      any capital 'P's. This is a kludge.
+#    (mimetype contains "application/<OTHER>")
+>>50           string  !epub+zip
+>>>50          string  !vnd.oasis.opendocument.
+>>>>50         string  !vnd.sun.xml.
+>>>>>50                string  !vnd.kde.
+>>>>>>38       regex   [!-OQ-~]+               Zip data (MIME type
"%s"?)
 !:mime  application/zip
->>4    byte            0x0b            Zip archive data, at least v1.1
to extract
-!:mime application/zip
->>0x161        string          WINZIP          Zip archive data, WinZIP
self-extracting
-!:mime application/zip
->>4    byte            0x14            Zip archive data, at least v2.0
to extract
+#    (mimetype contents other than "application/*")
+>26            string  \x8\0\0\0mimetype
+>>38           string  !application/
+>>>38          regex   [!-OQ-~]+               Zip data (MIME type
"%s"?)
 !:mime  application/zip
 
-# EPUB (OEBPS) books using OCF (Open Container Format)
-# From: Adam Buchbinder <adam.buchbinder at gmail.com>
-# URL: http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm, section 4.
->>38   string  application/epub+zip    EPUB ebook
-!:mime application/epub+zip
-
-# EPUB documents (.epub eBooks) are also zip archives starting with a
-# 'mimetype' member, which contains 'application/epub+zip'.
->>50   string          epub+zip        epub eBook data
-!:mime application/epub+zip
-
-# Catch other ZIP-with-mimetype formats
->>50           string          !epub+zip
->>>50          string          !vnd.oasis.opendocument.
->>>>50         string          !vnd.sun.xml.
->>>>>50                string          !vnd.kde.
->>>>>>38       regex           [!-OQ-~]+       Zip data (mimetype
"%s"?)
->26    string          \x8\0\0\0mimetype
->>38   string          !application/
->>>38  regex           [!-OQ-~]+               Zip data (mimetype
"%s"?)
-
+# Generic zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
+#   Next line excludes specialized formats:
+>26    string          !\x8\0\0\0mimetype      Zip archive data
+!:mime application/zip
+>>4    byte            0x09            \b, at least v0.9 to extract
+>>4    byte            0x0a            \b, at least v1.0 to extract
+>>4    byte            0x0b            \b, at least v1.1 to extract
+>>0x161        string          WINZIP          \b, WinZIP
self-extracting
+>>4    byte            0x14            \b, at least v2.0 to extract
 
 # Zoo archiver
 20      lelong          0xfdc4a7dc      Zoo archive data
@@ -740,7 +736,7 @@
 
 # GTKtalog catalogs
 # by David Necas (Yeti) <yeti at physics.muni.cz>
-4      string  gtktalog\       GTKtalog catalog data,
+4      string  gtktalog\       GTKtalog catalog data,
 >13	string	3		version 3
 >>14	beshort	0x677a		(gzipped)
 >>14	beshort	!0x677a		(not gzipped)
@@ -757,12 +753,12 @@
 !:mime  application/x-bittorrent
 
 # Atari MSA archive - Teemu Hukkanen <tjhukkan at iki.fi>
-0       beshort 0x0e0f          Atari MSA archive data
->2      beshort x               \b, %d sectors per track
->4      beshort 0               \b, 1 sided
->4      beshort 1               \b, 2 sided
->6      beshort x               \b, starting track: %d
->8      beshort x               \b, ending track: %d
+0      beshort 0x0e0f          Atari MSA archive data
+>2     beshort x               \b, %d sectors per track
+>4     beshort 0               \b, 1 sided
+>4     beshort 1               \b, 2 sided
+>6     beshort x               \b, starting track: %d
+>8     beshort x               \b, ending track: %d
 
 # Alternate ZIP string (amc at arwen.cs.berkeley.edu)
 0       string  PK00PK\003\004  Zip archive data
@@ -806,8 +802,8 @@
 >>0x2A	string	>\0		: %s
 
 # DR-DOS 7.03 Packed File *.??_
-0      string  Packed\ File\   Personal NetWare Packed File
->12    string  x               \b, was "%.12s"
+0      string  Packed\ File\   Personal NetWare Packed File
+>12    string  x               \b, was "%.12s"
 
 # EET archive
 # From: Tilman Sauerbeck <tilman at code-monkey.de>


From cchittleborough+comp.file at cluemail.com  Tue Oct 26 09:51:10 2010
From: cchittleborough+comp.file at cluemail.com (Christopher Chittleborough)
Date: Tue, 26 Oct 2010 17:21:10 +1030
Subject: Web-accessible source repository?
Message-ID: <1288075870.26377.1401963809@webmail.messagingengine.com>

A couple of us have submitted patches against released versions which
failed because
of changes since the released versions. Assuming the sources are kept in
a repository,
would it be possible to give read-only web access to that repository?
That would
make it a little easier to submit patches.

Cheers -- Chris Chittleborough


From jkaluza at redhat.com  Tue Oct 26 10:17:41 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 26 Oct 2010 09:17:41 +0200
Subject: Web-accessible source repository?
In-Reply-To: <1288075870.26377.1401963809@webmail.messagingengine.com>
References: <1288075870.26377.1401963809@webmail.messagingengine.com>
Message-ID: <201010260917.41702.jkaluza@redhat.com>

On Tuesday, October 26, 2010 08:51:10 am Christopher Chittleborough wrote:
> A couple of us have submitted patches against released versions which
> failed because
> of changes since the released versions. Assuming the sources are kept in
> a repository,
> would it be possible to give read-only web access to that repository?
> That would
> make it a little easier to submit patches.

I think that's really great idea, because you would also know if your patch is 
accepted or not and you would not have to track accepted patches on mailing 
list.

> Cheers -- Chris Chittleborough
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Regards,
Jan Kaluza


From jpinon at olfeo.com  Tue Oct 26 10:31:39 2010
From: jpinon at olfeo.com (Julien Pinon)
Date: Tue, 26 Oct 2010 09:31:39 +0200
Subject: Incorrect mime info for windows executables
In-Reply-To: <0784D725-E328-4942-A092-BE5C549723FD@palisadesys.com>    (Guy
	Helmer's message of "Mon, 25 Oct 2010 23:17:02 +0200")
References: <83r5fifh5q.fsf@cookie.olfeo-lab.net>
	<0784D725-E328-4942-A092-BE5C549723FD@palisadesys.com>
Message-ID: <83y69ll89w.fsf@cookie.olfeo-lab.net>

Le lundi 25 octobre 2010 ? 23:17,
Guy Helmer <ghelmer at palisadesys.com> ?crivait :

> On Oct 22, 2010, at 9:14 AM, Julien Pinon wrote:
>
>> Hi,
>>
>> I allways  get application/octet-stream  mime type for  some dos/windows
>> executables with any 5-0x release.

> This is the change I am using (against the magic files from 5.03) for DOS/Windows executables.
>
> Guy

Thanks, it applies cleanly on 5.04 and resolves my problem.
-- 
Julien


From christos at zoulas.com  Tue Oct 26 17:08:10 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 26 Oct 2010 10:08:10 -0400
Subject: Patch to magdir/archive: better handling of ZIP-based formats
In-Reply-To: <1288075328.23702.1401962711@webmail.messagingengine.com>
	from "Christopher Chittleborough" (Oct 26,  5:12pm)
Message-ID: <20101026140810.2595256426@rebar.astron.com>

On Oct 26,  5:12pm, cchittleborough+comp.file at cluemail.com ("Christopher Chittleborough") wrote:
-- Subject: Patch to magdir/archive: better handling of ZIP-based formats

| Here's a patch to magic/magdir/archive (1.58 2010/09/20 19:19:16)
| which improves the handling of ZIP and derived formats. It makes the
| rules a little more robust, fixes some minor problems, removes some
| duplication and adds explanatory comments. You can also get it from
| http://cchittleborough.cluemail.com/archive.full.patch if necessary.
| 
| This patch also tidies up whitespace in some unrelated rules. If you
| don't want this,
| http://cchittleborough.cluemail.com/archive.minimal.patch
| is an alternate version of the patch which only touches ZIP-related
| rules.
| 
| 
| --- file-CVS/magic/magdir/archive       2010-10-25 03:41:15.000000000
| +0000
| +++ file-NEW/magic/Magdir/archive       2010-10-26 06:08:11.000000000
| +0000
| @@ -276,7 +276,7 @@
|  # Splint
|  0       string  \x93\xb9\x06 Splint archive data
|  # InstallShield
| -0      string   \x13\x5d\x65\x8c InstallShield Z archive Data
| +0      string  \x13\x5d\x65\x8c InstallShield Z archive Data
|  # Gather
|  1       string  GTH Gather archive data
|  # BOA
| @@ -497,7 +497,7 @@
|  0       string          HPAK            HPACK archive data
|  
|  # JAM Archive volume format, by Dmitry.Kohmanyuk at UA.net
| -0      string          \351,\001JAM\           JAM archive,
| +0      string          \351,\001JAM\           JAM archive,
|  >7	string		>\0			version %.4s
|  >0x26	byte		=0x27			-
|  >>0x2b	string          >\0			label %.11s,
| @@ -569,13 +569,17 @@
|  # Zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
|  0       string          PK\003\004
|  
| -# Specialised zip formats
| ->30    string          mimetype
| +# Specialised zip formats which start with a member named 'mimetype'
| +# (stored uncompressed, with no 'extra field') containing the file's
| MIME type.
| +# Check for have 8-byte name, 0-byte extra field, name "mimetype", and
| +#  contents starting with "application/":
| +>26    string          \x8\0\0\0mimetypeapplication/
|  
| -# OpenOffice.org / KOffice / StarOffice documents
| -# From: Abel Cheung <abel at oaka.org>
| +#  KOffice / OpenOffice & StarOffice / OpenDocument formats
| +#    From: Abel Cheung <abel at oaka.org>
|  
| -# KOffice (1.2 or above) formats
| +#   KOffice (1.2 or above) formats
| +#    (mimetype contains "application/vnd.kde.<SUBTYPE>")
|  >>50	string	vnd.kde.		KOffice (>=1.2)
|  >>>58	string	karbon			Karbon document
|  >>>58	string	kchart			KChart document
| @@ -586,8 +590,8 @@
|  >>>58	string	kspread			KSpread document
|  >>>58	string	kword			KWord document
|  
| -# OpenOffice formats (for OpenOffice 1.x / StarOffice 6/7)
| -#  mimetype contains 'application/vnd.sun.xml.<SUBTYPE>'
| +#   OpenOffice formats (for OpenOffice 1.x / StarOffice 6/7)
| +#    (mimetype contains "application/vnd.sun.xml.<SUBTYPE>")
|  >>50	string	vnd.sun.xml.		OpenOffice.org 1.x
|  >>>62	string	writer			Writer
|  >>>>68	byte	!0x2e			document
| @@ -605,9 +609,9 @@
|  >>>62	string	math			Math document
|  >>>62	string	base			Database file
|  
| -# OpenDocument formats (for OpenOffice 2.x / StarOffice >= 8)
| -# http://lists.oasis-open.org/archives/office/200505/msg00006.html
| -#  mimetype contains 'application/vnd.oasis.opendocument.<SUBTYPE>'
| +#   OpenDocument formats (for OpenOffice 2.x / StarOffice >= 8)
| +#    http://lists.oasis-open.org/archives/office/200505/msg00006.html
| +#    (mimetype contains "application/vnd.oasis.opendocument.<SUBTYPE>")
|  >>50	string	vnd.oasis.opendocument.	OpenDocument
|  >>>73	string	text
|  >>>>77	byte	!0x2d			Text
| @@ -651,48 +655,40 @@
|  >>>>78	string	-template		Template
|  !:mime  application/vnd.oasis.opendocument.image-template
|  
| -# OEBPS Container Format (OCF) 1.0
| ->>50   string  epub+zip                OEBPS Container Format (OCF) 1.0
| (EPUB)
| +#  EPUB (OEBPS) books using OCF (OEBPS Container Format)
| +#    From: Adam Buchbinder <adam.buchbinder at gmail.com>
| +#    http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm, section 4.
| +#    (mimetype contains "application/epub+zip")
| +>>50   string  epub+zip                EPUB ebook data
|  !:mime  application/epub+zip
|  
| -# Generic zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
| -# N.B. Must come after specialised formats
| -# Next line simply matches anything, including "mimetype" as used above
| ->30    byte            x
| ->>4    byte            0x00            Zip archive data
| -!:mime application/zip
| ->>4    byte            0x09            Zip archive data, at least v0.9
| to extract
| -!:mime application/zip
| ->>4    byte            0x0a            Zip archive data, at least v1.0
| to extract
| +#  Catch other ZIP-with-mimetype formats
| +#      In a ZIP file, the bytes immediately after a member's contents
| are
| +#      always "PK". The 2 regex rules here print the "mimetype"
| member's
| +#      contents up to the first 'P'. Luckily, most MIME types don't
| contain
| +#      any capital 'P's. This is a kludge.
| +#    (mimetype contains "application/<OTHER>")
| +>>50           string  !epub+zip
| +>>>50          string  !vnd.oasis.opendocument.
| +>>>>50         string  !vnd.sun.xml.
| +>>>>>50                string  !vnd.kde.
| +>>>>>>38       regex   [!-OQ-~]+               Zip data (MIME type
| "%s"?)
|  !:mime  application/zip
| ->>4    byte            0x0b            Zip archive data, at least v1.1
| to extract
| -!:mime application/zip
| ->>0x161        string          WINZIP          Zip archive data, WinZIP
| self-extracting
| -!:mime application/zip
| ->>4    byte            0x14            Zip archive data, at least v2.0
| to extract
| +#    (mimetype contents other than "application/*")
| +>26            string  \x8\0\0\0mimetype
| +>>38           string  !application/
| +>>>38          regex   [!-OQ-~]+               Zip data (MIME type
| "%s"?)
|  !:mime  application/zip
|  
| -# EPUB (OEBPS) books using OCF (Open Container Format)
| -# From: Adam Buchbinder <adam.buchbinder at gmail.com>
| -# URL: http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm, section 4.
| ->>38   string  application/epub+zip    EPUB ebook
| -!:mime application/epub+zip
| -
| -# EPUB documents (.epub eBooks) are also zip archives starting with a
| -# 'mimetype' member, which contains 'application/epub+zip'.
| ->>50   string          epub+zip        epub eBook data
| -!:mime application/epub+zip
| -
| -# Catch other ZIP-with-mimetype formats
| ->>50           string          !epub+zip
| ->>>50          string          !vnd.oasis.opendocument.
| ->>>>50         string          !vnd.sun.xml.
| ->>>>>50                string          !vnd.kde.
| ->>>>>>38       regex           [!-OQ-~]+       Zip data (mimetype
| "%s"?)
| ->26    string          \x8\0\0\0mimetype
| ->>38   string          !application/
| ->>>38  regex           [!-OQ-~]+               Zip data (mimetype
| "%s"?)
| -
| +# Generic zip archives (Greg Roelofs, c/o zip-bugs at wkuvx1.wku.edu)
| +#   Next line excludes specialized formats:
| +>26    string          !\x8\0\0\0mimetype      Zip archive data
| +!:mime application/zip
| +>>4    byte            0x09            \b, at least v0.9 to extract
| +>>4    byte            0x0a            \b, at least v1.0 to extract
| +>>4    byte            0x0b            \b, at least v1.1 to extract
| +>>0x161        string          WINZIP          \b, WinZIP
| self-extracting
| +>>4    byte            0x14            \b, at least v2.0 to extract
|  
|  # Zoo archiver
|  20      lelong          0xfdc4a7dc      Zoo archive data
| @@ -740,7 +736,7 @@
|  
|  # GTKtalog catalogs
|  # by David Necas (Yeti) <yeti at physics.muni.cz>
| -4      string  gtktalog\       GTKtalog catalog data,
| +4      string  gtktalog\       GTKtalog catalog data,
|  >13	string	3		version 3
|  >>14	beshort	0x677a		(gzipped)
|  >>14	beshort	!0x677a		(not gzipped)
| @@ -757,12 +753,12 @@
|  !:mime  application/x-bittorrent
|  
|  # Atari MSA archive - Teemu Hukkanen <tjhukkan at iki.fi>
| -0       beshort 0x0e0f          Atari MSA archive data
| ->2      beshort x               \b, %d sectors per track
| ->4      beshort 0               \b, 1 sided
| ->4      beshort 1               \b, 2 sided
| ->6      beshort x               \b, starting track: %d
| ->8      beshort x               \b, ending track: %d
| +0      beshort 0x0e0f          Atari MSA archive data
| +>2     beshort x               \b, %d sectors per track
| +>4     beshort 0               \b, 1 sided
| +>4     beshort 1               \b, 2 sided
| +>6     beshort x               \b, starting track: %d
| +>8     beshort x               \b, ending track: %d
|  
|  # Alternate ZIP string (amc at arwen.cs.berkeley.edu)
|  0       string  PK00PK\003\004  Zip archive data
| @@ -806,8 +802,8 @@
|  >>0x2A	string	>\0		: %s
|  
|  # DR-DOS 7.03 Packed File *.??_
| -0      string  Packed\ File\   Personal NetWare Packed File
| ->12    string  x               \b, was "%.12s"
| +0      string  Packed\ File\   Personal NetWare Packed File
| +>12    string  x               \b, was "%.12s"
|  
|  # EET archive
|  # From: Tilman Sauerbeck <tilman at code-monkey.de>

Applied! Thanks.
BTW your MUA trashes text files....

christos


From christos at zoulas.com  Tue Oct 26 17:11:28 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 26 Oct 2010 10:11:28 -0400
Subject: Web-accessible source repository?
In-Reply-To: <1288075870.26377.1401963809@webmail.messagingengine.com>
	from "Christopher Chittleborough" (Oct 26,  5:21pm)
Message-ID: <20101026141128.D530956426@rebar.astron.com>

On Oct 26,  5:21pm, cchittleborough+comp.file at cluemail.com ("Christopher Chittleborough") wrote:
-- Subject: Web-accessible source repository?

| A couple of us have submitted patches against released versions which
| failed because
| of changes since the released versions. Assuming the sources are kept in
| a repository,
| would it be possible to give read-only web access to that repository?
| That would
| make it a little easier to submit patches.

Yes, I will work on that. Unfortunately I need to upgrade my boxes to
someone newer than 5 year old OS and hardware and I have not had the time.
In the meantime I think it is long overdue to release 5.05..

christos


From dennis.misc at gmail.com  Tue Oct 26 02:57:00 2010
From: dennis.misc at gmail.com (Dennis)
Date: Mon, 25 Oct 2010 19:57:00 -0400
Subject: docx, xlsx, pptx files
Message-ID: <AANLkTik1xW1Sifgx2cdrcUFnXsF+J1oes=dbGWhUxTmJ@mail.gmail.com>

Hi, I am looking for a way to get the file type of the compressed MS
files (with extensions docx, xlsx, and pptx).
I've downloaded and installed the latest version of file (5.04), and
it does not yet identify these file types, although it does a great
job of getting information about .doc..

Is there a plan in the future to support being able to distinguish
compressed MS file formats from other zip archive data?

If not, any suggestions on how this can be done.. on either Linux or
SunOS?  thanks for your help.


From allanc at chickenandporn.com  Tue Oct 26 19:10:38 2010
From: allanc at chickenandporn.com (Allan Clark)
Date: Tue, 26 Oct 2010 09:10:38 -0700
Subject: docx, xlsx, pptx files
In-Reply-To: <AANLkTik1xW1Sifgx2cdrcUFnXsF+J1oes=dbGWhUxTmJ@mail.gmail.com>
References: <AANLkTik1xW1Sifgx2cdrcUFnXsF+J1oes=dbGWhUxTmJ@mail.gmail.com>
Message-ID: <AANLkTinCKg1p0pDcJid8Oaa9VNGmOGpWab=EWHDVazZb@mail.gmail.com>

On Mon, Oct 25, 2010 at 16:57, Dennis <dennis.misc at gmail.com> wrote:
> Hi, I am looking for a way to get the file type of the compressed MS
> files (with extensions docx, xlsx, and pptx).
> I've downloaded and installed the latest version of file (5.04), and
> it does not yet identify these file types, although it does a great
> job of getting information about .doc..
>
> Is there a plan in the future to support being able to distinguish
> compressed MS file formats from other zip archive data?

This seems attainable by two methods:
1) decompress every possible archive to see what might be inside, or
2) ask the producer of these files to give them some additional
meta-data outside of the compression

#1 seems to take a variable amount of time (perhaps very very long for
many files) and would require effort to add this "look deeper" action
to the classification process; #2 seems to be less possible.

Dennis, you might look at the microsoft files to see if #2 is already
there to harvest.

Allan
-- 
allanc at chickenandporn.com? "??" http://linkedin.com/in/goldfish


From christos at zoulas.com  Tue Oct 26 19:29:33 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 26 Oct 2010 12:29:33 -0400
Subject: docx, xlsx, pptx files
In-Reply-To: <AANLkTik1xW1Sifgx2cdrcUFnXsF+J1oes=dbGWhUxTmJ@mail.gmail.com>
	from Dennis (Oct 25,  7:57pm)
Message-ID: <20101026162933.7509C56426@rebar.astron.com>

On Oct 25,  7:57pm, dennis.misc at gmail.com (Dennis) wrote:
-- Subject: docx, xlsx, pptx files

| Hi, I am looking for a way to get the file type of the compressed MS
| files (with extensions docx, xlsx, and pptx).
| I've downloaded and installed the latest version of file (5.04), and
| it does not yet identify these file types, although it does a great
| job of getting information about .doc..
| 
| Is there a plan in the future to support being able to distinguish
| compressed MS file formats from other zip archive data?
| 
| If not, any suggestions on how this can be done.. on either Linux or
| SunOS?  thanks for your help.

Yes, there is. We need to include an unzip library to do this reliably.

christos


From Jens.Schleusener at t-online.de  Tue Oct 26 19:46:17 2010
From: Jens.Schleusener at t-online.de (Jens Schleusener)
Date: Tue, 26 Oct 2010 18:46:17 +0200 (CEST)
Subject: Obvious "ASCII text" file detected as "HTML document text"
In-Reply-To: <20100321231645.217BE56425@rebar.astron.com>
References: <20100321231645.217BE56425@rebar.astron.com>
Message-ID: <alpine.LSU.2.02.1010261812000.15757@schiller.dip.t-dialin.net>

On Mon, 22 Mar 2010, Christos Zoulas wrote:

> On Mar 21,  5:52pm, ian at darwinsys.com (Ian Darwin) wrote:
> -- Subject: Re: Obvious "ASCII text" file detected as "HTML document text"
>
> | Yes, I'm quite familiar with the HTML document format, having been the
> | person
> | who released the world's first commercial HTML editor back in the early
> | 1990's.
> |
> | So. You want "file" to build in an XML/XHTML/HTML4 parser, so we can handle
> | DOCTYPEs, xmlns's, comments, spaces? Not gonna happen. Maybe a regex?
> |
> | \s?(<[^>]+>)+
> |
> | But it will fail if the user has a valid HTML file like
> | <html><head><title>This is not a tag</title>
> |
> | I don't know what the answer is, but it has to be something that doesn't
> | make file
> | even slower than it's already become.
> |
> | I'm guessing we'll end up having to write another C module to handle
> | HTML, XML, XHTML and related stuff. Thoughts?
>
> The problem is that browser will eat anything that remotely resembles
> html and try to render it. This has created a lot of pages that are not
> standards conformant, but nevertheless are displayed correctly in browsers.
>
> We went through approximately 50000 samples of web pages in the latest
> version of file, and before the changes to the magic file we had a recognition
> rate of approximately 60%; after the changes we have 99%.
>
> Sure we can write another c module for html, but let's come up with a spec
> for it first and see if this cannot be implemented as magic.
>
> christos

Sorry for replying to this old thread but now I found an according "file" 
behaviour that may be a real bug:

"file" seems to mark "ASCII text" files as "HTML document text" if it 
found anywhere a beginning HTML tag valid in the HEAD section (and the 
HEAD tag itself) - but also if the HTML tag string isn't delimited by an 
appropriate delimiter like a space or a ">".

So "pathological" strings like for e.g.

  TITLE0<TITLE1

or

  if LENGTH<HEADLENGTH

embedded in an "arbitrary" ASCII text would let "file" evaluate that files 
errorneously as "HTML document text". Some tested BODY section HTML tags 
didn't force that behaviour.

Regards

Jens

P.S.: Can someone point me to a description how "HTML document text" 
detection of "file" currently works?


From j at hug.gs  Tue Oct 26 19:13:48 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 26 Oct 2010 09:13:48 -0700
Subject: Web-accessible source repository?
In-Reply-To: <20101026141128.D530956426@rebar.astron.com>
References: <1288075870.26377.1401963809@webmail.messagingengine.com>
	<20101026141128.D530956426@rebar.astron.com>
Message-ID: <AANLkTi=0O_jsK5CydLzmN=Pqw_-3aWaqEJoOQBvU4-=J@mail.gmail.com>

On Tue, Oct 26, 2010 at 7:11 AM, Christos Zoulas <christos at zoulas.com> wrote:
> On Oct 26, ?5:21pm, cchittleborough+comp.file at cluemail.com ("Christopher Chittleborough") wrote:
> -- Subject: Web-accessible source repository?
>
> | A couple of us have submitted patches against released versions which
> | failed because
> | of changes since the released versions. Assuming the sources are kept in
> | a repository,
> | would it be possible to give read-only web access to that repository?
> | That would
> | make it a little easier to submit patches.
>
> Yes, I will work on that. Unfortunately I need to upgrade my boxes to
> someone newer than 5 year old OS and hardware and I have not had the time.
> In the meantime I think it is long overdue to release 5.05..

Is there anything we can do to help?  I have plenty of hardware I
could donate to the cause if you want, and I'm sure other people here
have resources to share too.


From daxim at cpan.org  Tue Oct 26 23:39:30 2010
From: daxim at cpan.org (Lars =?utf-8?b?RMmq4bSH4bSE4bSL4bSP4bShIOi/quaLieaWrw==?=)
Date: Tue, 26 Oct 2010 22:39:30 +0200
Subject: Web-accessible source repository?
In-Reply-To: <20101026141128.D530956426@rebar.astron.com>
References: <20101026141128.D530956426@rebar.astron.com>
Message-ID: <201010262239.30463.daxim@cpan.org>

I suggest converting to Git <http://git-scm.com/> and uploading the repo to 
GitHub <https://github.com/>. These tools make collaborating and merging so 
much easier and more fun than the process over the mailing list.

PS: I want to direct your attention to my message and patch at 
<http://mx.gw.com/pipermail/file/2010/000548.html> which apparently fell under 
the table.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mx.gw.com/pipermail/file/attachments/20101026/649cf19a/attachment.bin>

From j at hug.gs  Wed Oct 27 00:05:40 2010
From: j at hug.gs (Dr. Jesus)
Date: Tue, 26 Oct 2010 14:05:40 -0700
Subject: Web-accessible source repository?
In-Reply-To: <201010262239.30463.daxim@cpan.org>
References: <20101026141128.D530956426@rebar.astron.com>
	<201010262239.30463.daxim@cpan.org>
Message-ID: <AANLkTi=Tz-2rrhqeOOYEenJvV5ZVCD9p7uwMoUH3KXB3@mail.gmail.com>

On Tue, Oct 26, 2010 at 1:39 PM, Lars D?????? ??? <daxim at cpan.org> wrote:
> I suggest converting to Git <http://git-scm.com/> and uploading the repo to
> GitHub <https://github.com/>. These tools make collaborating and merging so
> much easier and more fun than the process over the mailing list.

I agree, github would probably satisfy the SCM needs of the file project.


From shaneearnshaw at gmail.com  Wed Nov  3 11:12:40 2010
From: shaneearnshaw at gmail.com (Shane Earnshaw)
Date: Wed, 3 Nov 2010 19:42:40 +1030
Subject: Incorrect mime info for windows executables
Message-ID: <AANLkTi=HAfmNSmgZWD8Kak1a+2FeGGjwyiOH5pYj4OT7@mail.gmail.com>

On Oct 25,  4:17pm, ghelmer at palisadesys.com
<http://mx.gw.com/mailman/listinfo/file> (Guy Helmer) wrote:
-- Subject: Re: Incorrect mime info for windows executables

|* On Oct 22, 2010, at 9:14 AM, Julien Pinon wrote:
*|*
*|* > Hi,
*|* >
*|* > I allways  get application/octet-stream  mime type for  some dos/windows
*|* > executables with any 5-0x release. For instance with putty.exe
*|* > (http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe
<http://the.earth.li/%7Esgtatham/putty/latest/x86/putty.exe>) :
*|* >
*|* > with libmagic 4-26:
*|* >
*|* > vz-3-12415:/tmp# file putty.exe
*|* > putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
*|* >
*|* > vz-3-12415:/tmp# file -i putty.exe
*|* > putty.exe: application/x-dosexec
*|* >
*|* >
*|* > with libmagic 5-04:
*|* >
*|* > vz-2-12415:/tmp# file putty.exe
*|* > putty.exe: PE32 executable for MS Windows (GUI) Intel 80386 32-bit
*|* >
*|* > vz-2-12415:/tmp# file -i putty.exe
*|* > putty.exe: application/octet-stream; charset=binary
*|* >
*|* >
*|* > I was  not able to  find if this  is a known problem  or not. Do  I miss
*|* > something ?
*|* >
*|*
*|*
*|* This is the change I am using (against the magic files from 5.03)
for DOS/Windows executables.
*|*

*On Oct 26, 1:49am, christos wrote:

> The patch does not apply cleanly because this file has been changed. Can
> you try to apply your patch on the following?
>
> Thanks,
>
> christos



Hi,

I've been looking at this exact problem within the last week or two.  My
initial thought was to edit the magic file as per Guy's suggestion, but was
worried that this would be fixing the issue with the DOS executable but not
for every case.  Unless the intent of the design is for there to be a mime
defined for each sub-level.

Looking at the source code, for 5.04, it appears that the mime will only be
printed if the top level has a description.  In the case of a DOS
executable, and probably many others, it does not have a description.  I'm
referring to the following line in the magic file.

 0                 string                 MZ

In the 4.26 source code a macro was used to determine if the description or
mime type was defined when printing a result, but from 5.0x only the
description is checked.  That aside, I believe that if the following patch
is applied to softmagic.c the problem is resolved for all cases.  The
handle_annotation function will only return 1 or -1 if the flags MAGIC_APPLE
or MAGIC_MIME_TYPE have been set and a mime type has been defined.  So in
the normal case where these flags aren't set then it will always return 0
and continue on as normal.  If MAGIC_MIME_TYPE is set, the mime type will be
printed, if defined, and return 1, which will cause the magic function to
return 1 and finish matching.

I have tested this on 40 different file types and found that using the same
magic files application/x-dosexec and application/x-java-applet are now
correctly identified.  There were no changes to the output if no flags were
set.  I have only done limited testing and haven't checked every possible
file and flag combination.  I assume Christos will likely have some other
tests.



--- softmagic.c    2009-10-19 23:40:20.000000000 +1030
+++ my_softmagic.c    2010-11-03 18:38:03.773437500 +1030
@@ -168,6 +168,9 @@
             continue;
         }

+        if ((e = handle_annotation(ms, m)) != 0)
+            return e;
+
         /*
          * If we are going to print something, we'll need to print
          * a blank before we print something else.
@@ -175,8 +178,6 @@
         if (*m->desc) {
             need_separator = 1;
             printed_something = 1;
-            if ((e = handle_annotation(ms, m)) != 0)
-                return e;
             if (print_sep(ms, firstline) == -1)
                 return -1;
         }
@@ -251,13 +252,15 @@
                     ms->c.li[cont_level].got_match = 0;
                     break;
                 }
+
+                if ((e = handle_annotation(ms, m)) != 0)
+                    return e;
+
                 /*
                  * If we are going to print something,
                  * make sure that we have a separator first.
                  */
                 if (*m->desc) {
-                    if ((e = handle_annotation(ms, m)) != 0)
-                        return e;
                     if (!printed_something) {
                         printed_something = 1;
                         if (print_sep(ms, firstline)



I apologise if the patch doesn't apply to your code, this is the first time
I've posted a fix for open source code.


Regards,

Shane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mx.gw.com/pipermail/file/attachments/20101103/5822fffc/attachment.html>

From christos at zoulas.com  Wed Nov  3 16:39:19 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 3 Nov 2010 10:39:19 -0400
Subject: Incorrect mime info for windows executables
In-Reply-To: <AANLkTi=HAfmNSmgZWD8Kak1a+2FeGGjwyiOH5pYj4OT7@mail.gmail.com>
	from Shane Earnshaw (Nov  3,  7:42pm)
Message-ID: <20101103143919.1E33356426@rebar.astron.com>

On Nov 3,  7:42pm, shaneearnshaw at gmail.com (Shane Earnshaw) wrote:
-- Subject: Re: Incorrect mime info for windows executables

| Hi,
| 
| I've been looking at this exact problem within the last week or two.  My
| initial thought was to edit the magic file as per Guy's suggestion, but was
| worried that this would be fixing the issue with the DOS executable but not
| for every case.  Unless the intent of the design is for there to be a mime
| defined for each sub-level.
| 
| Looking at the source code, for 5.04, it appears that the mime will only be
| printed if the top level has a description.  In the case of a DOS
| executable, and probably many others, it does not have a description.  I'm
| referring to the following line in the magic file.
| 
|  0                 string                 MZ
| 
| In the 4.26 source code a macro was used to determine if the description or
| mime type was defined when printing a result, but from 5.0x only the
| description is checked.  That aside, I believe that if the following patch
| is applied to softmagic.c the problem is resolved for all cases.  The
| handle_annotation function will only return 1 or -1 if the flags MAGIC_APPLE
| or MAGIC_MIME_TYPE have been set and a mime type has been defined.  So in
| the normal case where these flags aren't set then it will always return 0
| and continue on as normal.  If MAGIC_MIME_TYPE is set, the mime type will be
| printed, if defined, and return 1, which will cause the magic function to
| return 1 and finish matching.
| 
| I have tested this on 40 different file types and found that using the same
| magic files application/x-dosexec and application/x-java-applet are now
| correctly identified.  There were no changes to the output if no flags were
| set.  I have only done limited testing and haven't checked every possible
| file and flag combination.  I assume Christos will likely have some other
| tests.

I think this is the right fix so I have applied it.

Thanks!

christos


From christos at zoulas.com  Wed Nov  3 16:41:51 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 3 Nov 2010 10:41:51 -0400
Subject: Incorrect mime info for windows executables
In-Reply-To: <AANLkTi=HAfmNSmgZWD8Kak1a+2FeGGjwyiOH5pYj4OT7@mail.gmail.com>
	from Shane Earnshaw (Nov  3,  7:42pm)
Message-ID: <20101103144151.B8BBE56427@rebar.astron.com>

On Nov 3,  7:42pm, shaneearnshaw at gmail.com (Shane Earnshaw) wrote:
-- Subject: Re: Incorrect mime info for windows executables

| Hi,
| 
| I've been looking at this exact problem within the last week or two.  My
| initial thought was to edit the magic file as per Guy's suggestion, but was
| worried that this would be fixing the issue with the DOS executable but not
| for every case.  Unless the intent of the design is for there to be a mime
| defined for each sub-level.
| 
| Looking at the source code, for 5.04, it appears that the mime will only be
| printed if the top level has a description.  In the case of a DOS
| executable, and probably many others, it does not have a description.  I'm
| referring to the following line in the magic file.
| 
|  0                 string                 MZ
| 
| In the 4.26 source code a macro was used to determine if the description or
| mime type was defined when printing a result, but from 5.0x only the
| description is checked.  That aside, I believe that if the following patch
| is applied to softmagic.c the problem is resolved for all cases.  The
| handle_annotation function will only return 1 or -1 if the flags MAGIC_APPLE
| or MAGIC_MIME_TYPE have been set and a mime type has been defined.  So in
| the normal case where these flags aren't set then it will always return 0
| and continue on as normal.  If MAGIC_MIME_TYPE is set, the mime type will be
| printed, if defined, and return 1, which will cause the magic function to
| return 1 and finish matching.
| 
| I have tested this on 40 different file types and found that using the same
| magic files application/x-dosexec and application/x-java-applet are now
| correctly identified.  There were no changes to the output if no flags were
| set.  I have only done limited testing and haven't checked every possible
| file and flag combination.  I assume Christos will likely have some other
| tests.

Now that I think about it again, perhaps I spoke too soon. The empty
description is used so that file does not do early matching of rules
allowing further tests to determine if entry is supposed to be used.
This allows for more complex matching rules. I think the right fix is
again to add mime types on all the rules that print instead.

christos


From jkaluza at redhat.com  Thu Nov 18 13:33:52 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Thu, 18 Nov 2010 12:33:52 +0100
Subject: [PATCH] Separated text and binary patterns
Message-ID: <201011181233.52371.jkaluza@redhat.com>

Hi, 

attached patch fixes few magic patterns as described in previous thread [1]. 
The goal is to use text patterns only for "text files" (for example 
bash/python/perl scripts, html...) and binary patterns only for "binary files".

The benefit is that newly added magic patterns for text formats like "unified 
diff" or better Python magic pattern can use !:strength to beat more common 
patterns like:

0 string a                                                                                                                                                                                    
>3 string >\0 %s script text executable

The patch uses string/t directive as described in thread [1].

[1] http://mx.gw.com/pipermail/file/2010/000637.html

Thanks for response
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-text-patterns.patch
Type: text/x-patch
Size: 17420 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101118/4a894a70/attachment.bin>

From christos at zoulas.com  Tue Nov 23 00:49:56 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 22 Nov 2010 17:49:56 -0500
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <201011181233.52371.jkaluza@redhat.com>
	from Jan Kaluza (Nov 18, 12:33pm)
Message-ID: <20101122224956.97CE956426@rebar.astron.com>

On Nov 18, 12:33pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: [PATCH] Separated text and binary patterns

| Hi, 
| 
| attached patch fixes few magic patterns as described in previous thread [1]. 
| The goal is to use text patterns only for "text files" (for example 
| bash/python/perl scripts, html...) and binary patterns only for "binary files".

I think that the only one that think is wrong is:

| -0	search/1	FiLeStArTfIlEsTaRt	binscii (apple ][) text
| +0	search/1/b	FiLeStArTfIlEsTaRt	binscii (apple ][) text

Because this is really a text file. The only thing I am worried about is
html files that contain non-ascii characters. Will those fail the text test?

christos


From jkaluza at redhat.com  Tue Nov 23 11:44:15 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 23 Nov 2010 10:44:15 +0100
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <20101122224956.97CE956426@rebar.astron.com>
References: <20101122224956.97CE956426@rebar.astron.com>
Message-ID: <201011231044.15937.jkaluza@redhat.com>

On Monday, November 22, 2010 11:49:56 pm Christos Zoulas wrote:
> On Nov 18, 12:33pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: [PATCH] Separated text and binary patterns
> 
> | Hi,
> | 
> | attached patch fixes few magic patterns as described in previous thread
> | [1]. The goal is to use text patterns only for "text files" (for example
> | bash/python/perl scripts, html...) and binary patterns only for "binary
> | files".
> 
> I think that the only one that think is wrong is:
> | -0	search/1	FiLeStArTfIlEsTaRt	binscii (apple ][) text
> | +0	search/1/b	FiLeStArTfIlEsTaRt	binscii (apple ][) text
> 
> Because this is really a text file. The only thing I am worried about is
> html files that contain non-ascii characters. Will those fail the text
> test?

I wil try it and let you know.

I was more thinking about moving all HTML detection into File binary (as it's 
already defined in names.h), because just now File has problems with detection 
of C/C++/Java code which contains html. That's caused by a fact, that html 
patterns are too weak and File checks them before checks defined in names.h 
(it's variation of string patterns vs binary patterns problem...).

What do you think about it?

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Regards, 
Jan Kaluza


From christos at zoulas.com  Tue Nov 23 16:03:34 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 23 Nov 2010 09:03:34 -0500
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <201011231044.15937.jkaluza@redhat.com>
	from Jan Kaluza (Nov 23, 10:44am)
Message-ID: <20101123140334.E6EBB564C4@rebar.astron.com>

On Nov 23, 10:44am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: [PATCH] Separated text and binary patterns

| On Monday, November 22, 2010 11:49:56 pm Christos Zoulas wrote:
| > On Nov 18, 12:33pm, jkaluza at redhat.com (Jan Kaluza) wrote:
| > -- Subject: [PATCH] Separated text and binary patterns
| > 
| > | Hi,
| > | 
| > | attached patch fixes few magic patterns as described in previous thread
| > | [1]. The goal is to use text patterns only for "text files" (for example
| > | bash/python/perl scripts, html...) and binary patterns only for "binary
| > | files".
| > 
| > I think that the only one that think is wrong is:
| > | -0	search/1	FiLeStArTfIlEsTaRt	binscii (apple ][) text
| > | +0	search/1/b	FiLeStArTfIlEsTaRt	binscii (apple ][) text
| > 
| > Because this is really a text file. The only thing I am worried about is
| > html files that contain non-ascii characters. Will those fail the text
| > test?
| 
| I wil try it and let you know.
| 
| I was more thinking about moving all HTML detection into File binary (as it's 
| already defined in names.h), because just now File has problems with detection 
| of C/C++/Java code which contains html. That's caused by a fact, that html 
| patterns are too weak and File checks them before checks defined in names.h 
| (it's variation of string patterns vs binary patterns problem...).
| 
| What do you think about it?

I think that the "name" tests in file are weak (and many people remove them)
since they are only used as the last resort. It is also hard to write heuristic
algorithms like that in c, but I like to be proven wrong :-)


christoss


From jkaluza at redhat.com  Tue Nov 23 16:34:35 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 23 Nov 2010 15:34:35 +0100
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <20101123140334.E6EBB564C4@rebar.astron.com>
References: <20101123140334.E6EBB564C4@rebar.astron.com>
Message-ID: <201011231534.35578.jkaluza@redhat.com>

On Tuesday, November 23, 2010 03:03:34 pm Christos Zoulas wrote:
> On Nov 23, 10:44am, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: [PATCH] Separated text and binary patterns
> 
> | On Monday, November 22, 2010 11:49:56 pm Christos Zoulas wrote:
> | > On Nov 18, 12:33pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> | > -- Subject: [PATCH] Separated text and binary patterns
> | > 
> | > | Hi,
> | > | 
> | > | attached patch fixes few magic patterns as described in previous
> | > | thread [1]. The goal is to use text patterns only for "text files"
> | > | (for example bash/python/perl scripts, html...) and binary patterns
> | > | only for "binary files".
> | > 
> | > I think that the only one that think is wrong is:
> | > | -0	search/1	FiLeStArTfIlEsTaRt	binscii (apple ][) text
> | > | +0	search/1/b	FiLeStArTfIlEsTaRt	binscii (apple ][) text
> | > 
> | > Because this is really a text file. The only thing I am worried about
> | > is html files that contain non-ascii characters. Will those fail the
> | > text test?
> | 
> | I wil try it and let you know.
> | 
> | I was more thinking about moving all HTML detection into File binary (as
> | it's already defined in names.h), because just now File has problems
> | with detection of C/C++/Java code which contains html. That's caused by
> | a fact, that html patterns are too weak and File checks them before
> | checks defined in names.h (it's variation of string patterns vs binary
> | patterns problem...).
> | 
> | What do you think about it?
> 
> I think that the "name" tests in file are weak (and many people remove
> them) since they are only used as the last resort. It is also hard to
> write heuristic algorithms like that in c, but I like to be proven wrong
> :-)

OK, so I believe the proper solution of the problem described above is to 
write better patterns for those languages and use "strength" to put them 
before html.

I will try to write some patterns and test them with my regression tests.

> 
> christoss
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Thanks for response,
Jan Kaluza


From christos at zoulas.com  Tue Nov 23 16:39:03 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 23 Nov 2010 09:39:03 -0500
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <201011231534.35578.jkaluza@redhat.com>
	from Jan Kaluza (Nov 23,  3:34pm)
Message-ID: <20101123143903.F07E7564C4@rebar.astron.com>

On Nov 23,  3:34pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: [PATCH] Separated text and binary patterns

| OK, so I believe the proper solution of the problem described above is to 
| write better patterns for those languages and use "strength" to put them 
| before html.
| 
| I will try to write some patterns and test them with my regression tests.

Right, I think so. In theory the magic patterns should be good enough to
do those tests, otherwise we should be improving them to be good enough.
Hardcoding new binary tests should only be done when it is impossible to
describe the test as a magic pattern.

christos


From jkaluza at redhat.com  Tue Nov 23 19:25:55 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 23 Nov 2010 18:25:55 +0100
Subject: [PATCH] LUKS UUIDs truncated
Message-ID: <201011231825.55399.jkaluza@redhat.com>

Hi,

File doesn't print whole UUID for LUKS and LVM devices (see 
https://bugzilla.redhat.com/show_bug.cgi?id=656395 for more details).

The fix is to increase MAXString definition from 32 bytes to 48 bytes. The true 
is it makes compiled magic files little bit bigger, but I think it should not 
problem these days.

Regards,
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-string-size.patch
Type: text/x-patch
Size: 672 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101123/0717d49c/attachment.bin>

From christos at zoulas.com  Tue Nov 23 23:06:05 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 23 Nov 2010 16:06:05 -0500
Subject: [PATCH] LUKS UUIDs truncated
In-Reply-To: <201011231825.55399.jkaluza@redhat.com>
	from Jan Kaluza (Nov 23,  6:25pm)
Message-ID: <20101123210605.C23E4564C4@rebar.astron.com>

On Nov 23,  6:25pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: [PATCH] LUKS UUIDs truncated

| Hi,
| 
| File doesn't print whole UUID for LUKS and LVM devices (see 
| https://bugzilla.redhat.com/show_bug.cgi?id=656395 for more details).
| 
| The fix is to increase MAXString definition from 32 bytes to 48 bytes. The true 
| is it makes compiled magic files little bit bigger, but I think it should not 
| problem these days.
| 
| Regards,
| Jan Kaluza

Bumped it to 64 bytes and the version to 8.

christos


From jkaluza at redhat.com  Thu Nov 25 11:41:39 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Thu, 25 Nov 2010 10:41:39 +0100
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <20101122224956.97CE956426@rebar.astron.com>
References: <20101122224956.97CE956426@rebar.astron.com>
Message-ID: <201011251041.39770.jkaluza@redhat.com>

On Monday, November 22, 2010 11:49:56 pm Christos Zoulas wrote:
> On Nov 18, 12:33pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: [PATCH] Separated text and binary patterns
> 
> | Hi,
> | 
> | attached patch fixes few magic patterns as described in previous thread
> | [1]. The goal is to use text patterns only for "text files" (for example
> | bash/python/perl scripts, html...) and binary patterns only for "binary
> | files".
> 
> I think that the only one that think is wrong is:
> | -0	search/1	FiLeStArTfIlEsTaRt	binscii (apple ][) text
> | +0	search/1/b	FiLeStArTfIlEsTaRt	binscii (apple ][) text
> 
> Because this is really a text file. The only thing I am worried about is
> html files that contain non-ascii characters. Will those fail the text
> test?

I have tested those patterns with various html files with non-ascii characters 
in different encodings (utf-8, utf-16, windows-1250, windows-1251...) and it 
works well for me.
 
> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Regards,
Jan Kaluza


From jerome.dupont at bnf.fr  Mon Nov 22 11:51:13 2010
From: jerome.dupont at bnf.fr (jerome.dupont at bnf.fr)
Date: Mon, 22 Nov 2010 10:51:13 +0100
Subject: [PATCH adding signature for arc files and a mimetype for warc file
Message-ID: <OF0E7E339D.E3097D01-ONC12577E3.00347640-C12577E3.003620B6@LocalDomain>


Hello ,

Here's a patch for /arc and warc file, which are format used to stored web
crawled datas.
2 modifications
_ mimetype added for warc format
_ creation of a new format: arc

Only
Regards,
J?r?me Dupont
Biblioth?que Nationale de France

Only  ./file-5.04/magic/Magdir/warc is modified:

diff  ./file-5.04/magic/Magdir/warc ./file-5.04patche/magic/Magdir/warc
--- ./file-5.04/magic/Magdir/warc       Sat Sep 19 18:28:13 2009
+++ ./file-5.04patche/magic/Magdir/warc Tue Nov 16 14:18:14 2010
@@ -5,3 +5,12 @@

 0      string  WARC/   WARC Archive
 >5     string  x       version %.4s
+!:mime application/warc
+
+#------------------------------------------------------------------------------
+# Arc File Format from Internet Archive
+# see http://www.archive.org/web/researcher/ArcFileFormat.php
+0      string          filedesc://     Internet Archive File
+!:mime application/x-ia-arc
+>11    search/256      \x0A    \b
+>>&0   ubyte   >0      \b version %c
\ No newline at end of file


Exposition  La France de Raymond Depardon  - jusqu'au 9 janvier 2011 - BnF - Fran?ois-Mitterrand / Grande Galerie 
Avant d'imprimer, pensez ? l'environnement. 

From christos at zoulas.com  Thu Nov 25 16:53:46 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Thu, 25 Nov 2010 09:53:46 -0500
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <201011251041.39770.jkaluza@redhat.com>
	from Jan Kaluza (Nov 25, 10:41am)
Message-ID: <20101125145346.A8F13564C4@rebar.astron.com>

On Nov 25, 10:41am, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: Re: [PATCH] Separated text and binary patterns

| I have tested those patterns with various html files with non-ascii characters 
| in different encodings (utf-8, utf-16, windows-1250, windows-1251...) and it 
| works well for me.

Ok, I will commit it.

Thanks,

christos


From jerome.dupont at bnf.fr  Fri Nov 26 11:32:18 2010
From: jerome.dupont at bnf.fr (jerome.dupont at bnf.fr)
Date: Fri, 26 Nov 2010 10:32:18 +0100
Subject: [PATCH adding signature for arc files and a mimetype for warc file
Message-ID: <OFE6C04E3D.F20C45FE-ONC12577E7.00331EB4-C12577E7.00346568@LocalDomain>



Hello ,

Here's a patch for /arc and warc file, which are formats used to stored web
crawled datas.
2 modifications
_ mimetype added for warc format
_ creation of a new format: arc

Regards,
J?r?me Dupont
Biblioth?que Nationale de France

Only  ./file-5.04/magic/Magdir/warc is modified:

diff  ./file-5.04/magic/Magdir/warc ./file-5.04patche/magic/Magdir/warc
--- ./file-5.04/magic/Magdir/warc       Sat Sep 19 18:28:13 2009
+++ ./file-5.04patche/magic/Magdir/warc Tue Nov 16 14:18:14 2010
@@ -5,3 +5,12 @@

 0      string  WARC/   WARC Archive
 >5     string  x       version %.4s
+!:mime application/warc
+
+#------------------------------------------------------------------------------
+# Arc File Format from Internet Archive
+# see http://www.archive.org/web/researcher/ArcFileFormat.php
+0      string          filedesc://     Internet Archive File
+!:mime application/x-ia-arc
+>11    search/256      \x0A    \b
+>>&0   ubyte   >0      \b version %c
\ No newline at end of file





Exposition  Primitifs de la photographie. Le calotype en France (1843 - 1860)  - jusqu'au 16 janvier 2011 - BnF - Richelieu / Galerie de photographie  Avant d'imprimer, pensez ? l'environnement. 

From jkaluza at redhat.com  Tue Dec  7 16:00:38 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Tue, 7 Dec 2010 15:00:38 +0100
Subject: [PATCH] Unified diff and better Python patterns
Message-ID: <201012071500.38758.jkaluza@redhat.com>

Hi,

attached are two patches. The first one adds support for unified diff (diffs 
generated by "diff -u"). The second one adds some patterns which detects Python 
scripts. Both are tested with my regression tests and should not bring any new 
regression.

Regards,
Jan Kaluza 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-unified-diff.patch
Type: text/x-patch
Size: 426 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101207/0ff8be25/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-python.patch
Type: text/x-patch
Size: 956 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101207/0ff8be25/attachment-0001.bin>

From christos at zoulas.com  Tue Dec  7 19:36:28 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 7 Dec 2010 12:36:28 -0500
Subject: [PATCH] Unified diff and better Python patterns
In-Reply-To: <201012071500.38758.jkaluza@redhat.com>
	from Jan Kaluza (Dec  7,  3:00pm)
Message-ID: <20101207173628.4260C56426@rebar.astron.com>

On Dec 7,  3:00pm, jkaluza at redhat.com (Jan Kaluza) wrote:
-- Subject: [PATCH] Unified diff and better Python patterns

| Hi,
| 
| attached are two patches. The first one adds support for unified diff (diffs 
| generated by "diff -u"). The second one adds some patterns which detects Python 
| scripts. Both are tested with my regression tests and should not bring any new 
| regression.
| 
| Regards,
| Jan Kaluza 

Thanks, applied!

christos


From jkaluza at redhat.com  Wed Dec 15 15:17:50 2010
From: jkaluza at redhat.com (Jan Kaluza)
Date: Wed, 15 Dec 2010 14:17:50 +0100
Subject: [PATCH] Separated text and binary patterns
In-Reply-To: <20101123143903.F07E7564C4@rebar.astron.com>
References: <20101123143903.F07E7564C4@rebar.astron.com>
Message-ID: <201012151417.50487.jkaluza@redhat.com>

On Tuesday, November 23, 2010 03:39:03 pm Christos Zoulas wrote:
> On Nov 23,  3:34pm, jkaluza at redhat.com (Jan Kaluza) wrote:
> -- Subject: Re: [PATCH] Separated text and binary patterns
> 
> | OK, so I believe the proper solution of the problem described above is to
> | write better patterns for those languages and use "strength" to put them
> | before html.
> | 
> | I will try to write some patterns and test them with my regression tests.
> 
> Right, I think so. In theory the magic patterns should be good enough to
> do those tests, otherwise we should be improving them to be good enough.
> Hardcoding new binary tests should only be done when it is impossible to
> describe the test as a magic pattern.

Hi,

Attached patches move C/C++/PO recognition from names.h into particular magic 
files. I have also tweaked strength of some other patterns to sort patterns 
list properly (so some weak html patterns have less strength than some better 
C++ patterns, Perl patterns have to be tried before C/C++ ones, ...).

This fixes many problems when C/C++ source codes are detected as HTML/SGML.

> christos
> 
> _______________________________________________
> File mailing list
> File at mx.gw.com
> http://mx.gw.com/mailman/listinfo/file

Regards,
Jan Kaluza
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-names.patch
Type: text/x-patch
Size: 1407 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101215/eee975dd/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.04-ascmagic-to-magic.patch
Type: text/x-patch
Size: 2875 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101215/eee975dd/attachment-0001.bin>

From james.hunt at ubuntu.com  Thu Dec 16 19:13:59 2010
From: james.hunt at ubuntu.com (James Hunt)
Date: Thu, 16 Dec 2010 17:13:59 +0000
Subject: [PATCH] magic update for linux swap
Message-ID: <20101216171359.GB4274@azul>

Attached patch adds new compressed swap signature to linux magic.

See:

https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/683605

Kind regards,

James
-- 

James Hunt
______________________________________________________________________
Ubuntu Foundations Team, Canonical.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: magic-update-linuxswap.patch
Type: text/x-diff
Size: 734 bytes
Desc: not available
URL: <http://mx.gw.com/pipermail/file/attachments/20101216/6760cf47/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mx.gw.com/pipermail/file/attachments/20101216/6760cf47/attachment-0001.bin>

From christos at zoulas.com  Fri Dec 17 16:18:51 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Fri, 17 Dec 2010 09:18:51 -0500
Subject: [PATCH] magic update for linux swap
In-Reply-To: <20101217085255.GA4170@azul> from James Hunt (Dec 17,  8:52am)
Message-ID: <20101217141851.45E7A564C4@rebar.astron.com>

On Dec 17,  8:52am, james.hunt at ubuntu.com (James Hunt) wrote:
-- Subject: [PATCH] magic update for linux swap


Thanks, I will CC: the list.

christos

| Hi Christos,
| 
| I am sending this patch direct as although I have joined the mailing
| list twice (yesterday), it seems my mails are not being accepted.
| 
| Would it be possible to add this mail to the list to allow me to
| reference the mailing list in the fix for the bug below?
| 
| Thanks in advance.
| 
| Kind regards,
| 
| James.
| 
| 
| ----- Forwarded message from James Hunt <james.hunt at ubuntu.com> -----
| 
| Date: Thu, 16 Dec 2010 17:13:59 +0000=0AFrom: James Hunt <james.hunt at ubuntu=
| .com>=0ATo: file at mx.gw.com=0ASubject: [PATCH] magic update for linux swap=
| =0A=0A=
| Attached patch adds new compressed swap signature to linux magic.
| 
| See:
| 
| https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/683605
| 
| Kind regards,
| 
| James
| - --=20
| 
| James Hunt
| ______________________________________________________________________
| Ubuntu Foundations Team, Canonical.
| 
| diff -ru file-5.04/magic/Magdir/linux file-5.04-modified/magic/Magdir/linux
| - --- file-5.04/magic/Magdir/linux	2009-10-23 17:44:59.000000000 +0100
| +++ file-5.04-modified/magic/Magdir/linux	2010-12-16 17:04:49.726507004 +00=
| 00
| @@ -57,6 +57,8 @@
|  # From: Jeff Bailey <jbailey at ubuntu.com>
|  # Linux swap file with swsusp1 image, from Jeff Bailey <jbailey at ubuntu.com>
|  4076	string		SWAPSPACE2S1SUSPEND	Linux/i386 swap file (new style) with SWS=
| USP1 image
| +# From: James Hunt <james.hunt at ubuntu.com>
| +4076    string          SWAPSPACE2LINHIB0001    Linux/i386 swap file (new =
| style) (compressed hibernate)
|  # according to man page of mkswap (8) March 1999
|  4086	string		SWAPSPACE2	Linux/i386 swap file (new style)
|  >0x400	long		x		%d (4K pages)
| 
| 
| 
| 
| ----- End forwarded message -----


From glen at delfi.ee  Tue Dec 21 12:58:17 2010
From: glen at delfi.ee (=?ISO-8859-1?Q?Elan_Ruusam=E4e?=)
Date: Tue, 21 Dec 2010 12:58:17 +0200
Subject: PHP WSDL Cache magic
Message-ID: <4D108849.5000609@delfi.ee>

hi

i've started composing magic for php wsdl cache

so far i've came up with:

http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/file/php-wsdl-cache.magic?rev=1.3

it would be perfect if magic format supported pstring with size of 
little endian 4-bytes, as the format is
lelong specifying string length.

0x0b 0x00 0x00 0x00 "http://test"

so it currently prints the string up to 32 chars (is that limitation 
changeable?) until it meets garbage.

how about supporting byte offset where to take string length,
the same way the parenthesized offsets are supported, i.e: 
"pstring/[bislBISL]"

so i could write it as:

 >10 lelong <0xff
 >>&-4 pstring/l >0 \b, uri: "%s"


-- 
glen


From christos at zoulas.com  Wed Dec 22 21:10:59 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Wed, 22 Dec 2010 14:10:59 -0500
Subject: PHP WSDL Cache magic
In-Reply-To: <4D108849.5000609@delfi.ee>
	from =?ISO-8859-1?Q?Elan_Ruusam=E4e?= (Dec 21, 12:58pm)
Message-ID: <20101222191059.C62EA56426@rebar.astron.com>

On Dec 21, 12:58pm, glen at delfi.ee (=?ISO-8859-1?Q?Elan_Ruusam=E4e?=) wrote:
-- Subject: PHP WSDL Cache magic

| hi
| 
| i've started composing magic for php wsdl cache
| 
| so far i've came up with:
| 
| http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/file/php-wsdl-cache.magic?rev=1.3
| 
| it would be perfect if magic format supported pstring with size of 
| little endian 4-bytes, as the format is
| lelong specifying string length.
| 
| 0x0b 0x00 0x00 0x00 "http://test"
| 
| so it currently prints the string up to 32 chars (is that limitation 
| changeable?) until it meets garbage.
| 
| how about supporting byte offset where to take string length,
| the same way the parenthesized offsets are supported, i.e: 
| "pstring/[bislBISL]"
| 
| so i could write it as:
| 
|  >10 lelong <0xff
|  >>&-4 pstring/l >0 \b, uri: "%s"

Try ftp://ftp.astron.com/pri/file-5.04.tar.gz
Note: this beta's python support is broken; it is supposed to work
on both 2.x and 3.x but I think it works in none. Any help is appreciated :-)


Happy Holidays,

christos


From christos at zoulas.com  Mon Dec 27 01:11:30 2010
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 26 Dec 2010 18:11:30 -0500
Subject: Patch that adds GObject-introspection magic
In-Reply-To: <7f7CUooBIMF0+WtuZeIcHeDnpjE@QsmfhJNucgI88DfvPJdT1/nyboE>
	from Eygene Ryabinkin (Dec 26, 11:47pm)
Message-ID: <20101226231131.08C8B56426@rebar.astron.com>

On Dec 26, 11:47pm, rea at freebsd.org (Eygene Ryabinkin) wrote:
-- Subject: Patch that adds GObject-introspection magic

| Binary blobs (.typelib) files, produced with gobject-introspection
| suite will be recognized by file utility if this patch will be
| included to the distribution.
| -- 
| Eygene Ryabinkin                                        ,,,^..^,,,
| [ Life's unfair - but root password helps!           | codelabs.ru ]
| [ 82FE 06BC D497 C0DE 49EC  4FF0 16AF 9EAE 8152 ECFB | freebsd.org ]

Applied, thanks!

christos