[14:20:35]  out of curiosity, why do we build without frame pointers on x86-64 ? [14:29:36]  halfline: -fno-omit-frame-pointer produces less effective code and with -fasynchronous-unwind-tables as being used it has no benefit for debugging purposes (as .eh_frame is present in the main binary rpm). [14:34:49]  are we building with -fasynchronous-unwind-tables by default now? [14:37:55]  ssp: do you know if sysprof can cope with -fasynchronous-unwind-tables as well as it does wth -fno-omit-frame-pointer ? [14:38:25]  (on x86-64 i mean) [14:38:26]  halfline: At the moment it completely ignores unwind tables [14:38:36]  ah [14:39:01]  I don't know what it means for them to be asynchronous [14:39:47]  ssp: apparently they aren't asynchronous [14:40:06]  they're just generated so they can be proccessed from asynchronous events [14:41:20]  The main problem with unwind tables for sysprof is that you can't get at them in kernel mode [14:41:20]  wlel the man page says the table is exact at each instruction boundary [14:41:22]  whatever that means [14:42:51]  -fasynchronous-unwind-tables means the tables are complete as you want for debugging.  don't worry about the name. [14:43:27]  if you are willing to just read the text from user memory, you can always get them that way [14:44:22]  roland: But it's unlikely to be in memory, so reading them would generate page faults [14:44:44]  And I need to read them from interrupt context [14:45:18]  Plus, disk accesses might skew the profiles [14:45:44]  ssp: right, no can do.  of course, anything sampling only user stuff that actually does its stuff in interrupt context is just designed wrong.  but it's the popular wrong. [14:46:14]  looks like we've had -fasynchronous-unwind-tables in $RPM_OPT_FLAGS since before f8 at least [14:46:24]  sysprof doesn't just sample user stuff ... [14:46:25]  sysprof can, on other arches, sample things in interrupt context as well. [14:46:45]  roland: Well, what's the alternative? [14:47:28]  ssp: systemtap has its own hell for this very issue, but it's in a way that requires you to know beforehand what user binaries you care about [14:47:38]  ssp: would letting sysprof just profile for longer mitigate the skew from faults ? [14:48:35]  roland: FWIW, the Solaris compiler defaults to frame pointers on x86-64 precisely because they need them for dtrace [14:49:02]  ssp: personally i think even stuff sampling both kernel and user should be organized so it only cares to find the user part of a backtrace from closer to user mode where faulting stuff in is ok.  but nothing dynamic like that can really avoid the faults polluting the performance data. [14:49:42]  solaris has so much more control of everything than we ever do in linux, and even if we did, the abi choice was made before we started [14:50:22]  roland: 'made before we started' <-- what does that mean? [14:50:29]  roland: Well, compiling with frame pointers by default would still be ABI compliant, just slightly less efficient [14:51:00]  ssp: do you know how less efficient? [14:51:15]  halfline: i meant the world of x86-64 linux abi was pretty much settled before we started writing systemtap [14:51:45]  roland: oh, but this isn't an abi change that has significant impact on compatibility is it? [14:52:02]  ssp: the point is that you can't write a tool that presumes frame pointers and is useful in the world of linux binaries that exist [14:53:16]  well there's always going to be builds that don't work well with a sepcific debug tool [14:54:12]  i guess the point is to get our builds working well with the debug tools [14:54:13]  roland: Right, but sysprof is all about getting the common case right. If gcc (even just the Fedora gcc) started generating frame pointers on x86-64, a lot of the time it would just work [14:54:21]  or getting the debug tools working well with our builds [14:54:26]  or somewhere in between [14:55:09]  or you could do userspace callouts to load the kernel .eh_frame sections and mlock them. [14:55:22]  this would be... distasteful.  but it could work. [14:56:09]  ajax: can you do that before profiling starts ? [14:56:23]  because i think the problem was doing it JIT will skew the profiles [14:56:41]  ajax: There is a number of things that could be done, but they are all somewhat insane in various ways [14:57:29]  i'm a bit surprised that amd64 is still considered sufficiently register-starved that omit-frame-pointers helps. [14:57:49]  yeah, well, we're silly.  we let the compiler people make the right optimization decisions and then don't degrade them to make writing debugging tools easier but instead work on making debugging tools work with optimal code.  solaris is big on hyping how fabulous their debugging tools are because they quietly constrain them to working in the easy cases by degrading the compilations. [14:57:49]  halfline: Well, that's the problem if you solve the much harder problem of actually taking the page faults in a sane way [14:59:35]  roland: note i'm not necessarily saying that omiting frame pointers is the wrong decision. i have no idea [14:59:42]  i'm just trying to understand the situation [14:59:53]  and hopefully come up with a solution for my sysprof issues [15:00:14]  halfline: i don't really know the numbers, i'm just defending the decision my team made because i'm sure they knew what they were doing ;-) [15:01:09]  halfline: in short, the userspace backtrace issue is an issue others also contend with.  feel free to noodle on systemtap@sourceware [15:02:23]  alright