notes on portable executable

Post by fir
this format is somewhat trashy so to say, like four
headers instead of one, space wasted bad names
those 4 headers are
- dos header
- dos stub
- coff small header
- coff optional header
then goes
- list of sections
and then
- sections

You've not gone into image directories yet then?

This stuff just goes on and on. At some point I might eventually find
the real data that I need. Below is the 'optional header' struct I've
just had to create (not in C). Apparently I have to use this to find
image directories than it goes on from there.

I've created such formats for my byte-code projects, which have similar
problems to solve, and they are vastly simpler by comparison.

Post by fir
normal format imo should contain only small header
(what info this header should contain i am not quite
sure, minimal os name? minimal cpu set? number and
size of stacks?) (its important to know that but i dont
know)

But what else would you expect from MS? A company worth hundreds of
$billions cannot be expected to come up with a toy file format even it
would work perfectly well.

--------------------------------------------------

type imagedir=struct
wt_dword virtualaddr
wt_dword size
end

type optionalheader=struct !exe/dll only
wt_word magic
byte majorlv
byte minorlv
wt_dword codesize
wt_dword idatasize
wt_dword zdatasize
wt_dword entrypoint
wt_dword codebase
! wt_dword database !32-bit files only
word64 imagebase
wt_dword sectionalignment
wt_dword filealignment
wt_word majorosv
wt_word minorosv
wt_word majorimagev
wt_word majorssv
wt_word minorssv
wt_dword win32version
wt_dword imagesize
wt_dword headerssize
wt_dword checksum
wt_word subsystem
wt_word dllcharacteristics
word64 stackreserve
word64 stackcommit
word64 heapreserve
word64 heapcommit
wt_dword loaderflags
wt_dword rvadims
imagedir exporttable
imagedir importtable
imagedir resourcetable
imagedir exceptiontable
imagedir certtable
imagedir basereloctable
imagedir debug
imagedir architecture
imagedir globalptr
imagedir loadconfigtable
imagedir boundimport
imagedir iat
imagedir delayimportdescr
imagedir clrheader
imagedir reserved
end

--
bartc

Robert Wessel

2017-04-30 17:50:17 UTC

You've not gone into image directories yet then?
This stuff just goes on and on. At some point I might eventually find
the real data that I need. Below is the 'optional header' struct I've
just had to create (not in C). Apparently I have to use this to find
image directories than it goes on from there.
I've created such formats for my byte-code projects, which have similar
problems to solve, and they are vastly simpler by comparison.

But what else would you expect from MS? A company worth hundreds of
$billions cannot be expected to come up with a toy file format even it
would work perfectly well.

(..snip..)

PE is a modest adaptation of COFF. And while ELF has mostly replaced
COFF in *nix, it's not really any simpler (it does clean up some
ugliness in COFF, but add more functionality as well).

But as usual, the notion that something may be intended to do
something other than just the one task immediately in front of *you*,
escapes you.

bartc

2017-04-30 20:09:50 UTC

Post by bartc
But what else would you expect from MS? A company worth hundreds of
$billions cannot be expected to come up with a toy file format even it
would work perfectly well.

PE is a modest adaptation of COFF. And while ELF has mostly replaced
COFF in *nix, it's not really any simpler (it does clean up some
ugliness in COFF, but add more functionality as well).
But as usual, the notion that something may be intended to do
something other than just the one task immediately in front of *you*,
escapes you.

What seems to escape some people is the possibility that some things
might be a little more complex than they need to be. Or even a lot more
complex.

But with the EXE format, for what purpose does it exist other than
loading it into memory and running it? It's got just one task to do!

Let's see: you've got a block of bytes representing the program code,
another for the initialised data (these two will have different
attributes), some relocation info, and a list of dynamic library
imports. A few other bits and pieces.

Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

--
bartc

fir

2017-04-30 20:22:03 UTC

Post by bartc
But what else would you expect from MS? A company worth hundreds of
$billions cannot be expected to come up with a toy file format even it
would work perfectly well.

What seems to escape some people is the possibility that some things
might be a little more complex than they need to be. Or even a lot more
complex.
But with the EXE format, for what purpose does it exist other than
loading it into memory and running it? It's got just one task to do!
Let's see: you've got a block of bytes representing the program code,
another for the initialised data (these two will have different
attributes), some relocation info, and a list of dynamic library
imports. A few other bits and pieces.
Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

pe is not eleborate it is just 'messy'/ ugly /badly designed (some fields unused, nearly all names are bad names etc) besides that this is not eleborete format

same or nearly the same things is about winapi - it is also not eleborate and generally has whats is needed (functional) but its just a bit badly designed / ugly /bad names

Robert Wessel

2017-05-01 17:33:39 UTC

Post by bartc
But what else would you expect from MS? A company worth hundreds of
$billions cannot be expected to come up with a toy file format even it
would work perfectly well.

A rough subset of the object module format works just fine for
executables. Why introduce a whole new format? Obviously there are
systems where the two are distinct - as well as systems where they're
not, so both approaches work.

Post by bartc
Let's see: you've got a block of bytes representing the program code,
another for the initialised data (these two will have different
attributes), some relocation info, and a list of dynamic library
imports. A few other bits and pieces.
Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

Even after you include debugging, resources, multiple ISAs, and all
that? I rather doubt it'll be much simpler. While a simplified
format would handle many cases, you then end up with a even worse
situation, where you now have to support two formats (and obviously
that's been done too). And if you want to produce just a simple,
basic, executable without any of the fancy features, you can
boilerplate most of the stuff in a PE/COFF.

bartc

2017-05-01 20:04:27 UTC

Post by bartc
But what else would you expect from MS? A company worth hundreds of
$billions cannot be expected to come up with a toy file format even it
would work perfectly well.

Even after you include debugging, resources, multiple ISAs, and all
that? I rather doubt it'll be much simpler.

I have a similar requirement when I need a single binary file
representing all byte-code modules of an application. A dump of such a
file is here: https://pastebin.com/XkZ769YG (for 'hello world').

The exact same binary can work on 32- or 64-bit, Linux or Windows, x86
or ARM. You can see that it is remarkably simple and uncluttered. And
there are no fixed width fields in the binary to be adhered to
precisely. The file has an underlying structure which is a linear stream
of ints, floats and strings, each as wide as it needs to be.

Post by Robert Wessel
While a simplified
format would handle many cases, you then end up with a even worse
situation, where you now have to support two formats (and obviously
that's been done too). And if you want to produce just a simple,
basic, executable without any of the fancy features, you can
boilerplate most of the stuff in a PE/COFF.

I'm still looking at that to see what is essential, and what can be left
out. This is for EXE; for object files, I can either use my own format,
or eliminate them completely, by compiling straight to EXE.

(Or I could eliminate even that, by compiling to executable memory not
to disk. But generating a standalone EXE would be handy.)

--
bartc

bartc

2017-05-01 21:51:24 UTC

[PE file format]

Post by bartc
Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

Even after you include debugging, resources, multiple ISAs, and all
that? I rather doubt it'll be much simpler.

It's difficult enough that I'm completely stuck at the minute. Maybe
somebody knows the answer. I need to locate the Import Table.

The Import Data Directory header, in the Optional Header, contains:

(virtualaddress = 0x8000, size = 2076)

The 0x8000 is an RVA. "The RVA is the address of the table relative to
the base address of the image when the table is loaded."

0x8000 is an offset of +32768. The EXE file is only 16KB in size. I'm
trying to locate the import info it points to, but I've no idea how to
turn that 0x8000 into the actual offset.

The data I want appears to be at 0x3400 in the file (I found it by
visually looking for the numbers reported by PEDUMP). But how do I get
from 0x8000 to 0x3400? The code size is 7680, the image base is 4194304,
the code base is 4096... none of the numbers add up!

This is why I detest having to deal with other people's over the top
file formats.

(I may actually forget trying to sort out this EXE format, and use my
own format, but requiring a special EXE stub to load and run it. I may
still manage to finish it long before I've sussed out the EXE!

I can understand now why I looked at PE in the 1990s, and decided not to
bother.)

--
bartc

fir

2017-05-01 22:48:23 UTC

Post by bartc
[PE file format]

Post by bartc
Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

Even after you include debugging, resources, multiple ISAs, and all
that? I rather doubt it'll be much simpler.

It's difficult enough that I'm completely stuck at the minute. Maybe
somebody knows the answer. I need to locate the Import Table.
(virtualaddress = 0x8000, size = 2076)
The 0x8000 is an RVA. "The RVA is the address of the table relative to
the base address of the image when the table is loaded."
0x8000 is an offset of +32768. The EXE file is only 16KB in size. I'm
trying to locate the import info it points to, but I've no idea how to
turn that 0x8000 into the actual offset.
The data I want appears to be at 0x3400 in the file (I found it by
visually looking for the numbers reported by PEDUMP). But how do I get
from 0x8000 to 0x3400? The code size is 7680, the image base is 4194304,
the code base is 4096... none of the numbers add up!
This is why I detest having to deal with other people's over the top
file formats.
(I may actually forget trying to sort out this EXE format, and use my
own format, but requiring a special EXE stub to load and run it. I may
still manage to finish it long before I've sussed out the EXE!
I can understand now why I looked at PE in the 1990s, and decided not to
bother.)

you get it wrong, i may explain some things to you (i understand most parts of
it by now)

firstly, you got pe headers (a bit crappy ones), in the case of my own power dll it
ends at 0x178

the full calculation where it ends is

int pe_header_offset = *((int*) &pe_image[0x3c]);

int size_of_image_file_header = 24;

short size_of_optional_header = *((short*) &pe_image[pe_header_offset+20]);

int sections_table_offset = pe_header_offset + size_of_image_file_header + size_of_optional_header;

the sections_table_offset points where it ends, 0x178 may be typical but it depends probably on the size of a dos stub

(you yourself counted it in your own code)

then you got array/list of section info;
each of such info record has 40 bytes

number of this section is yet stored in headers at

short number_of_sections = *( (short*) &pe_image[pe_header_offset+6]);

the sections in my power dll are as my code prints it

This is Firex by fir 2017

Inspecting green.fire.dll ( 285696 bytes )

pe header offset 128 (0x0080) sections table offset 376 (0x0178)

.text 167992 bytes 58% 16 R-XI- EXECUTABLE
.data 91276 bytes 31% 64 RW-I-
.rdata 4020 bytes 1% 16 R--I-
.eh_fram 852 bytes 0% 4 R--I-
.bss 110984784 bytes 64 RW--U
.edata 6504 bytes 2% 4 R--I-
.idata 3012 bytes 1% 4 RW-I-
.CRT 24 bytes 0% 4 RW-I-
.tls 32 bytes 0% 4 RW-I-
.reloc 8436 bytes 2% 4 R--I- DISCARDABLE

R - section is readable, W - writeable, X - executable, I - initialised, U - ununitialised

the .idata would besection with imports,
at byte 20 in this 40 byte record mentioned you got int that is an file position (counted from begining of the file) of this import section in my case it is 042a00 (big number as this is after .code and .data whhich are about 300kb here) and there it is

youre confused by those RVA but you may skip it at this moment, those way above suffice.. as to those numbers youre confuse i may say that after those headers mentioned and after those list of sections you got sections placed itself..
sections are aligned typically to 0x200 (so first is expected to be typically at 0x200 or 0x400 depending how list of sections is long) but those sections when loaded in ram are typically aligned to 0x1000 (which is 4096 and relates to ram page sizes) - those pages when loaded in ram are copied under this mentioned base adress, this base adress for exa is said to be 0x00400000 * and 0x10000000 for dll,
but for dll it will vary.. RVAa are related to those base adress AFAIK..
in files hovever the adresses uset are
just byte numpers in a file

* the base adress for .code is 0x00400000
but afaik the usual entry point for code is 0x00401000 (im not 100% sure hovever) thus RVA of this entry point would be 0x1000 and so on

bartc

2017-05-01 23:23:51 UTC

Post by bartc
It's difficult enough that I'm completely stuck at the minute. Maybe
somebody knows the answer. I need to locate the Import Table.
(virtualaddress = 0x8000, size = 2076)
The 0x8000 is an RVA. "The RVA is the address of the table relative to
the base address of the image when the table is loaded."
0x8000 is an offset of +32768. The EXE file is only 16KB in size. I'm
trying to locate the import info it points to, but I've no idea how to
turn that 0x8000 into the actual offset.
The data I want appears to be at 0x3400 in the file (I found it by
visually looking for the numbers reported by PEDUMP). But how do I get
from 0x8000 to 0x3400? The code size is 7680, the image base is 4194304,
the code base is 4096... none of the numbers add up!

Yes, I looked again, and the data the Data Directory refers to (the
Import Table), is the same data in the .idata section. And that has the
0x3400 offset I wanted!

But then, what's the Data Directory stuff all about? I will need to
generate EXEs, and may need to create at least the Import Table Data
Directory. I don't know if it would work without it. (If I zero out the
import table data dir header of a working exe, it crashes.)

--
bartc

fir

2017-05-01 23:36:30 UTC

Yes, I looked again, and the data the Data Directory refers to (the
Import Table), is the same data in the .idata section. And that has the
0x3400 offset I wanted!
But then, what's the Data Directory stuff all about? I will need to
generate EXEs, and may need to create at least the Import Table Data
Directory. I don't know if it would work without it. (If I zero out the
import table data dir header of a working exe, it crashes.)
--
bartc

well i will be doin the same (even wrote already on this back then, i need it for assembly - i wrote basic one but yet without imports generation) - so if i wrote it i may post a pices of code and explanation you could then just freely use it (im not sure where i will be doin that if maybe this week but maybe more in future)

as to those 'data directory' i didnt care for this, more focused on this full sections list

i treat this data dirctory as probably some redundant pieces of 'shortcuts'.. section names are not quite standarised
and this was probably a fixed though partially redundant shortcut to point the same section as .idata do

if you need to flush imports you probably need to fill both .idata idata antry on section list yet those pointers in data directory too, but ofr general understending of the format the section list and sections itself are more important imo

i will be reading toomorow how itself import section is build as this is also a bit complicated but may go thru that with no terrible troubles probably

Robert Wessel

2017-05-01 23:41:44 UTC

Yes, I looked again, and the data the Data Directory refers to (the
Import Table), is the same data in the .idata section. And that has the
0x3400 offset I wanted!
But then, what's the Data Directory stuff all about? I will need to
generate EXEs, and may need to create at least the Import Table Data
Directory. I don't know if it would work without it. (If I zero out the
import table data dir header of a working exe, it crashes.)

It's used for a quick find after the load to the section. As I
mentioned in my other message, I don't know why it couldn't be
computed by the load from the data in the section table.

fir

2017-05-01 23:56:40 UTC

Yes, I looked again, and the data the Data Directory refers to (the
Import Table), is the same data in the .idata section. And that has the
0x3400 offset I wanted!
But then, what's the Data Directory stuff all about? I will need to
generate EXEs, and may need to create at least the Import Table Data
Directory. I don't know if it would work without it. (If I zero out the
import table data dir header of a working exe, it crashes.)

It's used for a quick find after the load to the section. As I
mentioned in my other message, I don't know why it couldn't be
computed by the load from the data in the section table.

afaik linkers can name sections with their own names like .imports .idata .i .IMP etc and loader cant be sure whuch section holds imports naybe, esp as section flags also do not denote this so they made this data directories and it show only how ugly and badly designed this format is
(also for example code section info is also redundant, but not in this data directory but inheader fields itself,
+ also section flages denote which one is exacutable (there are two redundant (or look like redundant ;c) bits
for that ;cc

fir

2017-05-03 14:49:10 UTC

Yes, I looked again, and the data the Data Directory refers to (the
Import Table), is the same data in the .idata section. And that has the
0x3400 offset I wanted!

ps wait, you fond it?
i write my app and now i cannot find it
(speaking on exports though not imports)

Inspecting green.fire.dll ( 285696 bytes )

pe header offset 128 (0x0080) sections table offset 376 (0x0178)

.text 167992 B 58% 400 16 R-XI- EXECUTABLE
.data 91276 B 31% 29600 64 RW-I-
.rdata 4020 B 1% 3fc00 16 R--I-
.eh_fram 852 B 0% 40c00 4 R--I-
.bss 110984784 B 0 64 RW--U
.edata 6504 B 2% 41000 4 R--I-
.idata 3012 B 1% 42a00 4 RW-I-
.CRT 24 B 0% 43600 4 RW-I-
.tls 32 B 0% 43800 4 RW-I-
.reloc 8436 B 2% 43a00 4 R--I- DISCARDABLE

R - section is readable, W - writeable, X - executable, I - initialised, U - ununitialised

data_directory_offset f8
dd_exports_offset 6a1c000
dd_imports_offset 6a1e000
dd_resources_offset 0
dd_bind_iports_offset 0

the exports in file seem to be physically at 41000 (and .edata entry points that)
where dd_export_offset is reported to be 6a1c000 (which may be probably the RVA in ram after load, im not sure)

Post by bartc
But then, what's the Data Directory stuff all about? I will need to
generate EXEs, and may need to create at least the Import Table Data
Directory. I don't know if it would work without it. (If I zero out the
import table data dir header of a working exe, it crashes.)
--
bartc

fir

2017-05-05 23:52:48 UTC

Post by fir
* the base adress for .code is 0x00400000
but afaik the usual entry point for code is 0x00401000 (im not 100% sure hovever) thus RVA of this entry point would be 0x1000 and so on

i readed on this and now i know why is
that

the 0x40 0000 (0x400000 but i will probably write it with spaces to easier read) (which is 4MB) comes from historycal reasons like form windows 98 or maybe even before where some low area needed to be fixed to dos areas and shared by al processes - something like that
in windows now they now also put stack in this area (probably 0x23 0000 to 0x03 0000 but im not sure)

0x40 1000 comes just form that that sections in ram are typically aligned by 0x1000 (4KB) and where it is mapped under
0x40 0000 the first 'section' are pe headers copied there code is usually next so it is 0x40 1000

BTW hell i dont know what is up 0x8000 0000 and above as sytem dlls seems are loaded like 0x7000 0000 - 0x8000 0000

wonder where they place heap - as all static arrays are never created at runtime (which is funny but i was not conscious of) they probably can take all the place betwwwen last mapped section of an exe and the begining of the dll-area

however 2 things come to my mind - as far as i know dll can use static arrays (mine do and alocates even 100 MB of static) and this mean it can consume quite part of ram too, second thing dlls can be loaded at runtime so probably loading dll's and heap may gave some conflict sometimes

system dlls though seem not to use noiticable static arrays - maybe they dont use noticable ram or just use heap
too

what is loaded above 0x8000 0000 i dont know.. drvers or what?

Robert Wessel

2017-05-01 23:26:53 UTC

Post by bartc
[PE file format]

Post by bartc
Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

Even after you include debugging, resources, multiple ISAs, and all
that? I rather doubt it'll be much simpler.

Find the .idata section header in the sections table (right after the
PE/COFF header), and use the Raw Data Offset field.

IIRC, the RVA in the Optional Header is used for finding the base for
the JMPs that get patched by the loader *after* the section is loaded
into memory, although I'm not 100% sure why you couldn't just compute
it after the load. You should see that the idata section should get
specified to be loaded at +0x8000. That will be different from the
raw addresses since things like the BSS section will not actually have
space allocated in the executable file (so if you have 16KB of text,
12kb of data and 8kb of bss before your idata section, you'll be off
by ~8kb because the bss section won't actually take any space).

And it's been long enough that I don't remember if the raw addresses
count the MZ header or not.

Robert Wessel

2017-05-01 23:40:23 UTC

On Mon, 01 May 2017 18:26:53 -0500, Robert Wessel

Post by bartc
[PE file format]

Post by bartc
Doesn't sound too difficult. If someone were to create such a file
format from scratch, the chances are that it would be far less elaborate
than the PE32/PE32+/COFF version.

Even after you include debugging, resources, multiple ISAs, and all
that? I rather doubt it'll be much simpler.

Find the .idata section header in the sections table (right after the
PE/COFF header), and use the Raw Data Offset field.
IIRC, the RVA in the Optional Header is used for finding the base for
the JMPs that get patched by the loader *after* the section is loaded
into memory, although I'm not 100% sure why you couldn't just compute
it after the load. You should see that the idata section should get
specified to be loaded at +0x8000. That will be different from the
raw addresses since things like the BSS section will not actually have
space allocated in the executable file (so if you have 16KB of text,
12kb of data and 8kb of bss before your idata section, you'll be off
by ~8kb because the bss section won't actually take any space).
And it's been long enough that I don't remember if the raw addresses
count the MZ header or not.

BTW, I assume you have the current MS PE Doc? And have you seen:

https://msdn.microsoft.com/en-us/library/ms809762.aspx

bartc

2017-05-01 23:59:34 UTC

Post by Robert Wessel
BTW, I assume you have the current MS PE Doc?

Yes, I have the 70-page .docx document. (Would have been nicer in PDF,
but then maybe with .docx I can delete the stuff that is not relevant
and get a shorter summary.)

--
bartc

Scott Lurndal

2017-05-02 14:01:06 UTC

Post by bartc
(I may actually forget trying to sort out this EXE format, and use my
own format, but requiring a special EXE stub to load and run it. I may
still manage to finish it long before I've sussed out the EXE!

COFF was obsolete when Microsoft started using it. You should
look at the COFF replacement - the Extensible Linkage Format (ELF).

s***@casperkitty.com

2017-05-02 23:30:50 UTC

If you're trying to have something that can build code as fast as possible,
without much concern for how fast the code runs, that approach can allow for
true single-pass executable generation if your code starts by figuring out
how big it is and includes a list of fix-ups shortly before the end. Unless
an executable is going to be run many times, the small extra time needed to
apply fix-ups after loading the code will be less than the amount of time
that would be needed to use a conventional multi-pass assembler or link
stage.

bartc

2017-05-03 00:06:19 UTC

Post by s***@casperkitty.com

I'm sure I've done many similar things in the past, and fixup time was
essentially zero.

A current project can take a 23,000 line application, in 27 modules, and
not only do all the fixups needed, but compile it from source code too,
in as little as 10ms. [Source to byte-code; figures vary with 10ms being
the fastest measured, but it will still be only tens of milliseconds.]

How long can it take to do fix-up a monolithic block of code, which has
already been compiled? (What luxury!)

External fixups such as linking to imported functions from shared
libraries, I will see if that can be done on-demand. (So if there are
100 functions but one run needs only 6, only those 6 are resolved.)

In any case if it's too slow, I will need to do something about it.

the small extra time needed to

Post by s***@casperkitty.com
apply fix-ups after loading the code will be less than the amount of time
that would be needed to use a conventional multi-pass assembler or link
stage.

The problem I had recently, which sparked this interest, was that a
program which took tens of milliseconds to compile from C to ASM, was
taking up to 90 seconds to assemble into object code. (I filed a bug
report as that assembler was slowing down to ridiculous speeds when the
input file got above a certain size (tens of Kloc). But I've had no
response.)

--
bartc

fir

2017-05-10 08:57:01 UTC

Post by bartc
I can understand now why I looked at PE in the 1990s, and decided not to
bother.)

now when i read on this pe format (recently slowed down a bit but still read into it)
i am sorry that i not learned it 16 years ago (in ninetees i probably dint know that c exist, but i learned fundaments of c and bits of winapi in about 2001 )

this is really important 'stuff' and i should learn it there.. i feel so much lagged

it is both fun and tangled but i hope ehen going yet thru that i will have bigger fun with this for example i will try to make a standard for encompassing function 'headers' in dll making in c code delclaratios not needed and yet probbaly i will also project own debug symbol standard usefull for debug/dissasembly - a drop of work but good thing

fir

2017-05-10 09:01:31 UTC