Attending bafug to discuss FreeBSD 6.1

Mike, Koji, and I went to the monthly Bay Area FreeBSD Users Group meeting tonight, and it was a fairly interesting, albeit low-key, meeting. The stated purpose of the meeting was to discuss the impending release of FreeBSD 6.1. The meeting was hosted by Julian Elischer, and we were joined by FreeBSD release guru, Scott Long, via phone. In general, I would say that the mood surrounding FreeBSD 6 (and especially 6.1) is very upbeat. What follows is the notes that I took, during Scott's talk and the general discussion that followed:

FreeBSD 5 vs. 6

• transition, not that 5 is bad, it was just transition
• some rough edges, in usability and performance of FS
• focus over last year has been on polishing and fixing bugs
• 6 is usable for desktops and servers and appliances
• 6 has proper threading support, with libthr. Works quite well in 6.1. Equivalent to pthreads at the moment. Two different libraries adhere to same API, so you can choose which one want to use, via libmap. Allows mapping different libraries to applications, on a per application basis. Similar to LD_PRELOAD.

Question about stability of 4.10 vs 6.0?

• empirically, stability of 6 is on par if not better than 4.10 -> some VM bugs in 4.8, 9, 10, got fixed in FreeBSD 4.11
• And of course, all of this stuff is fixed in FreeBSD 6.
• Definitely increased stability from 5 to 6.
• Definitely not worse than FreeBSD 4, some peripheral things that 4 might still be doing better (random USB device, soundcard, old ATA device)
• Been doing a lot more stress testing
• Yahoo! and Ironport are moving to FreeBSD 6
• hub.freebsd.org now running 6.1-PRERELEASE (12 cpu machine)
• 6 is working much better with multiple processors -- the threading is doing much better.

FreeBSD 6.1 is small features and fixing bugs:

• Have been back-porting features from -CURRENT into 6.1
• For example, tons of amd64 work from Yahoo! - can go 64bits, but keep custom 32bit applications running.
• Drivers added
• Large update to ATA disk driver, supports more software RAID, and some of the pseudo-hardware RAID
• SAS - Serial Attached SCSI support, same connector as SATA, but SCSI protocol - starting to get support in FreeBSD 6.1
• nullfs works much better now, can be used in jails

Upcoming talks:

• Nate Lawson talking about ACPI next month
• Nawaf Bitar, June bafug talk, was head of system development at SGI, giving talk on multithreading and multiprocessor schedulers.

My question: Are there any knobs or utilities for tweaking processor affinity? When changing Movable Type from BerkeleyDB to MySQL, I noticed (via top) that the perl process that was sucking from DB and writing to MySQL was competing for the same CPU that MySQL was using. To put it another way, it appeared as if both Perl and MySQL were running on the same CPU, when they should have been using different CPUs in my dual processor machine.

The response:

• What I saw could be a reporting error, with thread changes, been some reporting bugs concerning display of who is using what cpu
• things display as running on cpu 0, but might have been running on other processor, but when they release, they drive back to CPU 0
• should loadavg be 2 if you max out a dual processor box? (I can confirm that this was the case on FreeBSD 4.x, but I'll need to look to see if it is the case on 6.x).
• On FreeBSD, there is nothing to affect processor affinity. On "nice to do list".
• Julian was able to dredge up some knowledge of some some sysctls - kern.sched - some tunables, one called "followon" in particular. If you are a thread in single process, about to give up the CPU, find another thread from same process and continue (try to keep multiple threads on same cpu). This is turned off by default.
• The upcoming ULE scheduler might have some affinity stuff

I also asked what utilities I can use to determine what is really happening (if top can't be trusted). Julian responded:

• There is a kernel option, KTR, which enables fine-grained kernel tracing. It is different from ktrace or truss, in that KTR allows for an instrumentation view inside of the kernel.
• Lot of events have KTR flag on them, on compile time, compile in a mask of bits, then use a sysctl to activate them. logs entries into a KTR buffer in memory, can drop into kernel debugger and "show ktr" to show the list, or run a userland program to dump buffer out to file. ktrdump? One of the bits is for the scheduler, thousands of entries per second.
• Google for "KTR, scheduling, graph" -> utility to draw scheduling graph and show which thread on which CPU.

Interesting aside about ktrace:

• ktrace -p <pid> will attach to an existing pid, and print out the syscall information.
• ktrace -dp <pid> will attach to an existing pid, and all of its running children, and print out the aggregate syscall information.
• ktrace -dip <pid> does the same as above, but also captures information for children that spawn while ktrace is running.

Mike asked about CVSup vs csup:

• csup works for checking out a particular tag
• csup doesn't do mirroring of source tree
• csup has just been checked into current
• we think csup does as good as a job bandwidth-wise as CVSup

Random Notes:

• Doing FreeBSD 5.5 release to tide people over that can't move to 6 yet.
• HT is a performance gain if you are doing floating point work
• Xen virtualization will appear in 6.2, both domain 0 and and running in Xen VM. Each Xen VM will be able to claim a CPU, to better utilize multi-core boxes. Design of Xen require architectural things in the kernel that FreeBSD doesn't have yet.
• There was a question about the state of alternative schedulers in FreeBSD. The answer, via Scott, is that ULE is still fairly neglected, some work done a few months ago, still experimental. Not sure of the ETA.
• For wireless, there is something called "WPA supplement"(?), also does WEP, can also be used to configure wireless NIC before running dhclient
• rcorder - determine order that scripts will launch, based upon rc.conf and dependencies in the new RC system.

Whew! Like I said, I definitely learned some stuff, and I'm jazzed to try and play some more with my "so fresh and so clean" FreeBSD 6.1-PRERELEASE machine.

-Andy.

Technorati Tags: bafug, FreeBSD, UNIX, Open Source