LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 05-07-2013, 04:15 PM   #1
Simple12
LQ Newbie
 
Registered: Aug 2012
Posts: 16

Rep: Reputation: Disabled
Memory Content of 2 Identical Processes


Hi,

I have 2 identical processes running on Ubuntu doing the same thing (they have the same memory layout as well), from time to time I check on their memory content (at the same point of execution for both of them) to do memory content comparison. The issue is that I was expecting the memory content for the two processes to be the same till the end of execution however, they differ in some part of memory and I'm not able to figure out why!

Anyone have the idea why is that and if this is a normal situation?

I'm using memory map file and memory file of the processes to read their memory content and I'm using a python script to do the memory reading and checking.
 
Old 05-09-2013, 04:36 AM   #2
Sergei Steshenko
Senior Member
 
Registered: May 2005
Posts: 4,481

Rep: Reputation: 453Reputation: 453Reputation: 453Reputation: 453Reputation: 453
Quote:
Originally Posted by Simple12 View Post
Hi,

I have 2 identical processes running on Ubuntu doing the same thing (they have the same memory layout as well), from time to time I check on their memory content (at the same point of execution for both of them) to do memory content comparison. The issue is that I was expecting the memory content for the two processes to be the same till the end of execution however, they differ in some part of memory and I'm not able to figure out why!

Anyone have the idea why is that and if this is a normal situation?

I'm using memory map file and memory file of the processes to read their memory content and I'm using a python script to do the memory reading and checking.
Part of memory contents can be pointers, so why would one expect pointers to be the same ?
 
Old 05-09-2013, 07:23 AM   #3
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,125

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
Quote:
Originally Posted by Simple12 View Post
The issue is that I was expecting the memory content for the two processes to be the same till the end of execution however, they differ in some part of memory and I'm not able to figure out why!
Linux does some kinds of address randomization to make certain kinds of malware harder to make.

I don't know where the details are documented nor which specific kinds of address randomization are enabled in Linux.

A common and effective type of address randomization is to randomize (subject to a lot of constraints) the virtual address of the stack on process startup.

So OS's randomize the load position of DLLs or SOs. That is also common, but can be more destructive of performance with less benefit in malware immunity.

It is also possible to have some randomization of the allocations returned by malloc. That is a normal feature in Windows. I'm not sure about Linux. It drives me crazy in debugging Windows programs. I wish I knew some option to turn it off for debugging. You often need to restart the program you are debugging to get to a point slightly earlier in a chain of events. Without address randomization, data breakpoints would be an easy way to catch that earlier point in a chain of events. With address randomization, it can be almost impossible to backtrack a chain of events.

Quote:
Originally Posted by Sergei Steshenko View Post
Part of memory contents can be pointers, so why would one expect pointers to be the same ?
All those pointers are virtual addresses, not physical addresses. Without some sort of randomization, the virtual addresses should match.

Last edited by johnsfine; 05-09-2013 at 07:30 AM.
 
Old 05-09-2013, 07:39 AM   #4
Sergei Steshenko
Senior Member
 
Registered: May 2005
Posts: 4,481

Rep: Reputation: 453Reputation: 453Reputation: 453Reputation: 453Reputation: 453
Quote:
Originally Posted by johnsfine View Post
Linux does some kinds of address randomization to make certain kinds of malware harder to make.

I don't know where the details are documented nor which specific kinds of address randomization are enabled in Linux.

A common and effective type of address randomization is to randomize (subject to a lot of constraints) the virtual address of the stack on process startup.

So OS's randomize the load position of DLLs or SOs. That is also common, but can be more destructive of performance with less benefit in malware immunity.

It is also possible to have some randomization of the allocations returned by malloc. That is a normal feature in Windows. I'm not sure about Linux. It drives me crazy in debugging Windows programs. I wish I knew some option to turn it off for debugging. You often need to restart the program you are debugging to get to a point slightly earlier in a chain of events. Without address randomization, data breakpoints would be an easy way to catch that earlier point in a chain of events. With address randomization, it can be almost impossible to backtrack a chain of events.



All those pointers are virtual addresses, not physical addresses. Without some sort of randomization, the virtual addresses should match.
Code:
sergei@amdam2:~/junk> cat -n pointer_value.c
     1  #include <stdio.h>
     2
     3  static int i;
     4
     5  int main()
     6    {
     7    int j;
     8
     9    printf("address of i: %08x address of j: %08x\n", (unsigned)&i, (unsigned)&j);
    10    return 0;
    11    }
sergei@amdam2:~/junk> gcc -Wall -Wextra pointer_value.c
sergei@amdam2:~/junk> ./a.out
address of i: 0804a01c address of j: bfa89820
sergei@amdam2:~/junk> ./a.out
address of i: 0804a01c address of j: bffe4030
sergei@amdam2:~/junk> ./a.out
address of i: 0804a01c address of j: bfcd60e0
sergei@amdam2:~/junk> ./a.out
address of i: 0804a01c address of j: bfbdcfb0
sergei@amdam2:~/junk> ./a.out
address of i: 0804a01c address of j: bfea7290
sergei@amdam2:~/junk> ./a.out
address of i: 0804a01c address of j: bfa25be0
sergei@amdam2:~/junk>
- whatever.
 
Old 05-09-2013, 08:08 AM   #5
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,125

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
Quote:
Originally Posted by Sergei Steshenko View Post
whatever.
That is a nice simple demonstration of the randomization of the stack address.

As long as you are testing things (and I don't have root access to a Linux system at the moment), can you try that with randomization disabled?

I googled how to do that and found two answers

Code:
sysctl -w kernel.randomize_va_space=0
Code:
echo 0 > /proc/sys/kernel/randomize_va_space
I never tried either, so I don't know whether each of them works or just one or neither.

I also did a google search for an explanation of what the above commands do and found
http://docs.oracle.com/cd/E37670_01/...ernel_sec.html

Details might differ (but probably not much) in other distributions. That is just the one google found first.

Last edited by johnsfine; 05-09-2013 at 08:12 AM.
 
Old 05-09-2013, 08:33 AM   #6
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,425

Rep: Reputation: 1159Reputation: 1159Reputation: 1159Reputation: 1159Reputation: 1159Reputation: 1159Reputation: 1159Reputation: 1159Reputation: 1159
Realistically, two instances of the same process won't, for a variety of reasons, have an identical memory-layout. They might be "the same process," but they'll never be exactly-synchronized.

Linux does do intelligent things, however. It shares copies of code-segments, for example, although (for reasons described above and maybe others) they might not be in the same memory-location all the time.
 
Old 05-09-2013, 09:55 AM   #7
Simple12
LQ Newbie
 
Registered: Aug 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by johnsfine View Post
That is a nice simple demonstration of the randomization of the stack address.

As long as you are testing things (and I don't have root access to a Linux system at the moment), can you try that with randomization disabled?

I googled how to do that and found two answers

Code:
sysctl -w kernel.randomize_va_space=0
Code:
echo 0 > /proc/sys/kernel/randomize_va_space
I never tried either, so I don't know whether each of them works or just one or neither.

I also did a google search for an explanation of what the above commands do and found
http://docs.oracle.com/cd/E37670_01/...ernel_sec.html

Details might differ (but probably not much) in other distributions. That is just the one google found first.
I set the address randomization off using
Code:
 sysctl -w kernel.randomize_va_space=0
, and when I run the test code by Sergei Steshenko

Code:
     1  #include <stdio.h>
     2
     3  static int i;
     4
     5  int main()
     6    {
     7    int j;
     8
     9    printf("address of i: %08x address of j: %08x\n", (unsigned)&i, (unsigned)&j);
    10    return 0;
    11    }
I get the same address for variable j everytime I run the code however, I'm still getting a memory mismatch when I run the memory checking for the two processes. Any idea why?
 
Old 05-09-2013, 09:59 AM   #8
Simple12
LQ Newbie
 
Registered: Aug 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by sundialsvcs View Post
Realistically, two instances of the same process won't, for a variety of reasons, have an identical memory-layout. They might be "the same process," but they'll never be exactly-synchronized.

Linux does do intelligent things, however. It shares copies of code-segments, for example, although (for reasons described above and maybe others) they might not be in the same memory-location all the time.
even if I disable the address randomization?
 
Old 05-09-2013, 10:16 AM   #9
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,125

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
Quote:
Originally Posted by Simple12 View Post
I'm still getting a memory mismatch when I run the memory checking for the two processes. Any idea why?
I don't know what your process does.
A process might do something that explicitly depends on time.
A multi-threaded process can easily do things that have unpredictable sequence. If memory allocations are made during multi-threaded operations, those addresses might be non reproducible.

Quote:
Originally Posted by sundialsvcs View Post
Realistically, two instances of the same process won't, for a variety of reasons, have an identical memory-layout. They might be "the same process," but they'll never be exactly-synchronized.
Before address space randomization was standard, using single threaded non GUI programs, I had many occasions to depend on identical behavior (including identical addresses) for multiple instances of the same program with the same input. So I must disagree with your "never" claim. Unless there is a specific cause for the difference, there should not be a difference.

Last edited by johnsfine; 05-09-2013 at 10:20 AM.
 
Old 05-09-2013, 10:22 AM   #10
Simple12
LQ Newbie
 
Registered: Aug 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by johnsfine View Post
I don't know what your process does.
A process might do something that explicitly depends on time.
A multi-threaded process can easily do things that have unpredictable sequence. If memory allocations are made during multi-threaded operations, those addresses might be non reproducible.
I did the test on a process that does only printing numbers from 0 to 9

Code:
int  main(void)
{

int i;
for (i=0; i<10; i++)
printf("i= %d\n", i);

}
however still, running that program twice with two processes and testing their memory contents will give a mismatch.
 
Old 05-09-2013, 10:24 AM   #11
Simple12
LQ Newbie
 
Registered: Aug 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
just to note that I do memory checking at the same point of execution for both processes.
 
Old 05-09-2013, 10:38 AM   #12
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,125

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
Maybe something is wrong with the way you achieved "same point of execution".

Alternately, you didn't really start the processes identically. For example, usual ways of starting a process include copying all environment variables into that process's address space. Were any environment variables different?
 
Old 05-09-2013, 02:00 PM   #13
Simple12
LQ Newbie
 
Registered: Aug 2012
Posts: 16

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by johnsfine View Post
Maybe something is wrong with the way you achieved "same point of execution".

Alternately, you didn't really start the processes identically. For example, usual ways of starting a process include copying all environment variables into that process's address space. Were any environment variables different?

I didn't want to go into details but I will explain how I achieved this. I have my main program that will fork, then, in the new created process I call TRACEME and then execve with my program (the one I posted earlier as example). By now, I'm controlling the new process which has not yet started its execution (it's stopped until I continue its execution). After that I inject a fork system call to the new process to force it to duplicate itself and agian I force the new created process be traced by the main process. After that I will end up with two processes running the same program in which I'm tracing both of them.

Till here I have no problem. After that I start to run and intercept both processes at each system call, and from their I check their memory content.



I'm not sure if this will help, but I did intercept both processes at the instruction level (singlestep) (instead of the system call) to try to figure out where the differnce happend exactly, and I could tell at which instruction the change in memory happend.

after executing the instruction(hex) ff788589, the eax register values for both processes differ (each has different value, in the normal case the register values for both processes are the same all the time), till now the memory content is the same for both processes. Then 4d8bc031 instruction will execute (which I suspect to be a store instruction) and directly after executing that instruction I get the mismatch in memory content.

my problem is that I'm finding difficulties on translating the meaning of that instruction that caused the eax register of both processes to be differ.

Last edited by Simple12; 05-09-2013 at 02:08 PM.
 
  


Reply

Tags
linux, mem, memory, mmap, process


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
autofs local map files identical but not identical to automount jwaldram Linux - Server 2 10-26-2012 11:35 AM
top used memory vs ps processes memory a6speedo Linux - Newbie 7 06-13-2010 01:23 AM
Processes and memory? Itzuke Slackware 1 04-11-2006 02:57 PM
how 2 c the memory used by processes imppayel Linux - General 3 12-08-2004 08:40 AM
Memory And Processes smartes Linux - General 5 01-07-2003 03:02 PM


All times are GMT -5. The time now is 01:47 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration