Introduction

This week was spent attempting to debug the gccgo runtime via print statements. There were many things that I gained from this endeavour. The most significant of which, is the fact that I have got a great deal of information regarding the bootstrapping of a go process. Let’s proceed into presenting this week’s findings, shall we?

Findings

The process bootstrapping sequence

The code that begins a new go-process is conveniently located in a file called go-main.c, the most significant part of which is the following:

int
main (int argc, char **argv)
{
  runtime_check ();
  runtime_args (argc, (byte **) argv);
  runtime_osinit ();
  runtime_schedinit ();
  __go_go (mainstart, NULL);
  runtime_mstart (runtime_m ());
  abort ();
}

static void
mainstart (void *arg __attribute__ ((unused)))
{
  runtime_main ();
}

The process is as follows:

• First runtime_check runs and registers the os_Args and syscall_Envs as runtime_roots with the garbage collector. I am still investigating what this function exactly is doing, but it seems like some early initialisation of the garbage collector
• Secondly, runtime_args is run. It’s job is to call a specific argument handler for the arguments passed to main.
• Thirdly, runtime_osinit is run, whose job is to call the lowlevel _CPU_COUNT function, to get the number of CPUs (in a specific data structure that represents a set of CPUs)
• After that, runtime_schedinit is run, whose job is to create the very first goroutine (g) and system thread (m), and continues with parsing the command line arguments, and the environment variables. After that it sets the maximum number of cpus that are to be used (via GOMAXPROCS), runs the first goroutine, and does some last pieces of the scheduler’s initialisation.
• Following runtime_schedinit, __go_go is run, a function whose purpose is to create a new queue, tell it to execute the function that is passed to it as the first parameter, and then queue the goroutine in the global ready-to-run goroutine pool.
• Last but not least, runtime_mstart runs, which seems to be starting te execution of the kernel thread created during runtime_schedinit.

The very last piece of code that is run (and most probably the most important) is runtime_main. Remember that this is passed as a parameter to a goroutine created during the __go_go call, and its job is to mark the goroutine that called it as the main os thread, to initialise the sceduler, and create a goroutine whose job is to release unused memory (from the heap) back to the OS. It then starts executing the process user defined instructions (the code the programmer run) via a call to a macro that directs it to __go_init_main in the assembly generated by the compiler.

Runtime_main is also the function that terminates the execution of a go process, with a call to runtime_exit which seems to be a macro to the exit function.

Other findings

During our debugging sessions we found out that the total count of kernel threads that are running in a simple program is at least two. The first one is the bootstrap M, (the one initialised during the program’s initialisation, inside runtime_schedinit) and at least another one, (I am still invistigating the validity of the following claim) created to be used by the garbage collector.

A simple go program such as one doing arithmetic or printing a helloworld like message evidently has no issue running. The issues arrise when we use a go statement. With all our debugging messages activated, this is how a simple go program flows:

root@debian:~/Software/Experiments/go# ./a.out
[DEBUG] (in main) before runtime_mcheck is run
[DEBUG] (in main) before runtime_args is run
[DEBUG] (in main) before runtime_osinit is run
[DEBUG] (in main) before runtime_schedinit is run
[DEBUG] (in main) before runtime_mstart is run
[DEBUG] (in runtime_mstart) right before the call to runtime_minit
[DEBUG] (in mainstart) right before the call to runtime_main
[DEBUG] (in runtime_main) Beginning of runtime_main
[DEBUG] (start of runtime_newm) Total number of m's is 1
[DEBUG] (in runtime_newm) Preparing to create a new thread
[DEBUG] (in runtime_newm) Right before the call to pthread_create
[DEBUG] (in runtime_newm) pthread_create returned 0
[DEBUG] (in runtime_mstart) right before the call to runtime_minit
[DEBUG] (end of runtime_newm) Total number of m's is 2
Hello, fotis
[DEBUG] (in runtime_main) Right before runtime_exit

And this is how a goroutine powered program fails:

root@debian:~/Software/Experiments/go# ./a.out
[DEBUG] (in main) before runtime_mcheck is run
[DEBUG] (in main) before runtime_args is run
[DEBUG] (in main) before runtime_osinit is run
[DEBUG] (in main) before runtime_schedinit is run
[DEBUG] (in main) before runtime_mstart is run
[DEBUG] (in runtime_mstart) right before the call to runtime_minit
[DEBUG] (in mainstart) right before the call to runtime_main
[DEBUG] (in runtime_main) Beginning of runtime_main
[DEBUG] (start of runtime_newm) Total number of m's is 1
[DEBUG] (in runtime_newm) Preparing to create a new thread
[DEBUG] (in runtime_newm) Right before the call to pthread_create
[DEBUG] (in runtime_newm) pthread_create returned 0
[DEBUG] (in runtime_mstart) right before the call to runtime_minit
[DEBUG] (end of runtime_newm) Total number of m's is 2
[DEBUG] (start of runtime_new) Total number of m's is 2
[DEBUG] (in runtime_newm) Preparing to create a new thread.
[DEBUG] (in runtime_newm) Right before the call to pthread_create
a.out: ./pthread/pt-create.c:167: __pthread_create_internal: Assertion `({ mach_port_t ktid = __mach_thread_self (); int ok = thread->kernel_thread == ktid;
__mach_port_deallocate ((__mach_task_self + 0), ktid); ok; })' failed.
Aborted

Work for the next week

I will of course continue to print debug until I have knowledge of the exact flow of execution in the go runtime. Right now I have very good knowledge of the flow, but there are some things that I need to sort out. For instance it is not exactly clear to me why we call certain functions, or what they are supposed to be doing at certain parts. After I sort this out, I also plan to start debugging the libpthread to see what’s libpthreads status during a hello world like program, and during a goroutine powered program, to get to see if we get to find something interesting in libpthread (like how many threads does libpthread report against how many the goruntime reports)