Vadim Goncharov
2011-08-17 23:10:19 UTC
Hi,
I have bad news.
A month ago techlead of Rambler's Mail department declared in his blog
that they begin migration from FreeBSD to Debian/Ubuntu. Comments to the
blog entry (there was much flame) revealed that search department of
Russian search engine #1 Yandex.com also plans to migrate their
search cluster - about 30,000 servers in several DCs (~60% of total
their servers - to Linux. The problem here is that both big companies
were using FreeBSD from the beginning (~1997), so migration will be
rather expensive.
The official reasons (really semi-official, as these are individual
blogs), in short, were:
* inadequate package manager and huge monolithic base system
* lack of OpenVZ-like virtualization (need CPU limiting)
* a FreeBSD marketshare forecast for 5 years
Despite of several our committers working for them, the operating
expenses for FreeBSD are considered too high by these companies. They've
not confirmed, but the key problem is probably shortage of FreeBSD
specialists on the market to hire (e.g. Yandex has about 20 FreeBSD
admins and about 84 Linux admins, for all other their services and
40% of servers). So this is problem of FreeBSD's too low
userbase/marketshare, caused, in turn, by other FreeBSD's problems.
Given that they is just planning today, but not yet started, we have to
solve our problems or FreeBSD will effectively die[1]. Currently
https://ssl.netcraft.com/ssl-sample-report//CMatch/oscnt_all shows that
still 3% of servers are running FreeBSD. We are currently probably at
the tip of 'hype curve'[2] and then our userbase will begin to shrink,
and the task is to solve problems, so after hard time it will rise again
(hopefully more than it was before).
[1] 'Die' means shrinking userbase size to those of NetBSD/OpenBSD.
[2] http://www.metodolog.ru/01493/2.gif
Earlier this year in this list marcel@ have said about objective view:
http://lists.freebsd.org/pipermail/freebsd-arch/2011-January/010999.html
from the outside, not FreeBSD people. I have posted a call to Russian
community in my blog (http://nuclight.livejournal.com/128712.html) and
gathered a list of problems along with possible solutions to some of
them. The rest of this message is summary of them.
In general, people sympathizing FreeBSD agree that there are no fatal
technical problems and that FreeBSD could compete with mainstream Linux.
That said, the 80/20 principle is in effect and 80% losed users for
FreeBSD are caused by 20% problems. The trouble here is that what
"outside" people view as considerable problems, we here inside FreeBSD
view as something insignificant ("do it yourself"). Thus 80% could be
achieved by 20% of work. I have sorted them in order of decreased
importance (priority):
1. Social problems of community (and marketing, docs, ...)
2. Lack of drivers and virtualization.
3. Ports and packages.
4. Base system, closely related with packages.
5. Too short major releases' cycle.
6. Bug tracker, unicode and other less important trivia.
(bonus) FreeBSD strengths to be concentrated on.
Now all of them more detailed.
> 1. Social (psychologic) problems of community (marketing, docs, ...).
This is the most important one, because all technical problems are just
won't get solved because are even not viewed as problems. The FreeBSD
Project does not listen to users' needs. The typical response when poor
user want something is: "we don't need this, we won't change for you",
with "where are your patches?" at best. Then many users go out when see
such attitude toward them.
The key points are:
1) *The competent user is not zealot*.
2) The system is *for users, not for developers*.
With first: when user sees the system costs too much time for him, and
that those problems won't get fixed and even considered such - he will
switch to another OS. This may mean that he is able to follow our advice
hoe to do some or another thing (e.g. to recompile all ports), but this
is unacceptable to him because this is too much maintenance (another
system by objectivity requires less work). Many businesses fall into
this category. They will not with FreeBSD by any price just because they
love OS. Unlike ourselves.
With second: that's simple and not simple. It is simple in that by
nature each person don't make a work just for himself but for all
others, who are now "users" to him, too - just by induction that's not
for developers only. It is not simple due to question "Why do we need
those who don't contribute anything back to us?" which random committer
has right to ask. The answer, in short, is: there tasks which could be
only done by large groups of people, e.g. big corporations. For
corporations to invest and contribute back - the system must be popular
enough. To be popular enough means there must be a big amount of users
who is not contributing anything. It is useful in itself, though, that
some of those users, say 1%, will become contributors, that is, absolute
number of our developers will also benefit from FreeBSD being much more
popular.
There are opinions in our community like:
| > Personally I'd like to think that that we write an OS for users.
|
| We write an OS for the people who can and will use an OS written by
| us.
| FreeBSD is whatever its developers make it be
These are wrong and harmful nowadays. The world has changed. We have to
admit it or we will die. We have to admit mistakes and *change* some of
the ways we're doing things: any answer like "I don't agree, we don't
need these users/features/etc., we *won't* do this for someone" - is
just another step to grave.
Of course that doesn't mean to do everything - we often have not enough
time/resources to do. But that's another question and should not be
mixed with attitude - "it's good but we have no resources, will you do?"
vs "we don't need this, go away" (that's not only about features but
about keeping something too).
So ("system is for users") we need to (re)define who our user is. I view
the current situation as following:
100% .---------------. The stratification of overall
| Ubuntu | our userbase. Areas are:
| target | 6
| audience | 1 - official developers
expand -> |---------------|
to here | not our, but | 2 - actively participating and
| competent | 5 sending patches, contributors
| users | and other "zealots" (whose
|===============| "soul is with FreeBSD")
|announce@ only | 4
involved |---------------| 3 - not so active but discussing
| subscribed to | in mail lists, patches go
| our mailing | 3 sometimes
| lists |
committed |- - - - - - - -| 4 - busy with work to read lists
| | 2
|_______________| 5 - can use, but costs are now high
| *@freebsd.org | 1
0% `---------------' 6 - we'll never want them
The terms "involved" and "committed" were taken from
http://en.wikipedia.org/wiki/The_Chicken_and_the_Pig for those who are
"at the heart" of the community, at "periphery" and all others.
Currently, only those in (1) and partially (2) are listened to,
sometimes with "call for ..." in (3). Meanwhile, the most important of
our users, who are using FreeBSD in production, who are busy with real
work in real world, are in (4) and sometimes (5). Those are largely
ignored by the Project - even messages to announce@ last years are less
about important events and calls for volunteers/money.
The other social problem is lack of companies which offer commercial
support of FreeBSD like RedHat does.
As a possible solution to find out what user needs are, and to
compensate the fact that not all users in (4) and (5) care enough about
our principles and spirit, I propose a system for "weighted democracy".
This is a voting system (I could implement it), say, there is a mail
***@freebsd.org to which users send filled in vote forms, with
selected answers from a survey published in ***@.
The system has a database where users are recorded given their FreeBSD
activity in mail lists etc., and votes are summed as follows:
+ 1.00 vote for anyone not in database (not involved to FreeBSD)
+ a decimal logarithm of number of posts to mail lists (2 votes for 100
posts, 3 votes for 1000 posts)
+ a binary logarithm of num PRs in GNATS db (6 votes for 64 send-pr's)
+ a proportion (say, N/2) of entries for this person in "svn log" (e.g.
in "Submitted by:")
+ an assigned (by core@) number of votes in special exceptional DB (for
corporation and the like)
The system then presents results for each answer: 1) how many users
voted, 2) how many votes summed.
This is by no mean to measure "exact contribution", but to defend from
anonymous users and trolls who not care about amount of work developers
will need to. The results are viewed as a feedback to core team, not as
a final decision - that's all for what marketing is solely exist. There
no adequate feedback from users to developers currently at all -
individual posts in mail lists are too small for statistics (what about
ministat(1) for users, huh?..).
What is also in this part? Documentation. No, not in that sense. The
observations show that while there are described solutions to problems
in system, it's difficult for users to find it. It's somehow organized
or misorganized that solution is not intuitive for them (e.g. someone
migrating from Windows complained that it was difficult to find 'simply
how to format a flash drive' in Handbook). I don't know how to handle
this properly. May be catalogues with Best Practices howto's pointing to
exact sections of Handbook, FAQ, articles etc.? May be better search
mechanisms?..
And the last here about too much conservatism and rigidness in our camp.
As one of the opponents said (roughly translated):
| FreeBSD, historically, was good system, let's say, "valid system
| for solid people". Pureness, strictness, etc. The base system is
| consequence of this approach. The trouble is, that in 2011 nobody
| needs pureness and strictness. Customers need transparency and fast
| reaction to their requests. If there is no transparency and speed,
| you get CentOS 6 (just released, at last). Or FreeBSD (Gentoo,
| Solaris, you name it).
I don't agree that those certainly contradicts each other, but we
definitely need to change, er, something. And don't keep something just
because it is tradition, if that tradition has no more technical reasons
for our users. Let's shift conservatism to another field where it is
really needed (e.g. legacy support and other examples below).
Finally, two more quotes from the ***@freebsd.org archives of last
2 years:
| From: Cyrille Lefevre <cyrille.lefevre-***@laposte.net>
| [...] however, you answer remember me why I quit FreeBSD, cease
| fire, courtesy is the key
and
| From: Julian H. Stacey
| Though I & many others have sent lots of send-pr, contributing even
| a spelling correction to FreeBSD is much harder than to e.g.
| http://wikipedia.org
The threshold of entering to FreeBSD is too high and should be lowered.
I can confirm the last quote on my own example, 2 months ago. I've
submitted a patch to ipfw, adding call/return rule actions. It allows
to organize rules in somewhat like pf anchors, and, more importantly,
iptables chains, enabling FreeBSD ipfw to compete with Linux at least
partially in this sphere. The patch wasn't taken in maillists, I had
to push it in IRC, and push hard: the whole needed by many big users
thing was deferred (and still not MFCed to 8.x) just because some
#define's and printf's were Not So Proper & Right Way! F*cking shame.
> 2. Lack of drivers and poor guest virtualization.
It is known that FreeBSD supports less hardware than competitors. It is,
however, a matter of popularity - the more system will have users, the
more drivers will be created by 3rd parties, so we cannot directly
affect this. It's kind of exclusive circle.
But there is one area - virtualization - without supporting it the
FreeBSD niche will collapse very fast. It is necessary to say that it is
about guest (para-)virtualization only, not host mode. Host mode market
is already lost for FreeBSD for quite some time, may be something in the
future, but today it will waste of resources. What is needed here is
just analogue to OpenVZ (and jails/rctl already done more than half of
the work here).
And in guest mode FreeBSD *urgently needs* working drivers/utilities for
all common suits, Citrix XenServer, VMWare vSphere (ESXi), Xen*,
Microsoft Hyper-V, etc. This must be the primary direction for
developers' forces and FreeBSD Foundation's money. We have may be about
1 year here. Why? Because of cloud computing and VPS/VDS. It's no
matter what OS runs hoster if clients will use FreeBSD and we still have
users. I've been already asked by a customer for which I wrote
a non-portable kqueue-based daemon to rewrite for Linux because they
want to go for Amazon EC2 (it's cheaper as only used resources are
accounted) and FreeBSD there has still many in ISSUES scaring them...
> 3. Ports and packages.
What was the main problems with large-scale installations of FreeBSD in
that businesses? In short, that binary packages are not equal in rights
to ports, and that complicates things (i.e. requires too much work) when
one have many (> 10) servers. This was listed to me as:
1) No pkg and pkg-devel versions. The -devel version is headers, static
libs, programmer examples, etc. not needed in production (we could
say this part is what is actually depended on in B-deps).
2) Package name is dependent on options, so packages with another opts
don't work well when dependencies are rebuilt.
3) Conflicts: no way to have apache13 and apache22 the same time.
4) No dependence on base system. You may cut out something, recompile
world, deploy it on cluster and just then see that some packages are
now don't work.
5) Dependencies are badly designed. No version ranges in dependencies,
no alternative packages, no priorities in package search.
6) Update problems. The version is just coded into name of package, and
dependencies are on the entire name, so there are situations when
install/upgrade of just one package may require rebuild 3/4 of all
pkgs. You cannot easiy modify installed package without editing pkgdb
manually. It is impossible to upgrade/replace package by out of the
box tools.
7) Base system has no "out of the box" tools for package upgrade. Our
business opponents say this the least problem as one can always
install portupgrade, but conclude that overall base system concept
does not play well with full-featured packages (see also next part
about base system).
8) There is no -STABLE supported branches in ports.
All of this could be avoided (they know about tinderbox etc.) but just
requires too much work, for their basic tasks like automated upgrade of
entire system & packages or reinstall of needed packages.
That's problems. Next, possible (gathered) solutions.
It is obvious that current packages are not first-class citizens, in
comparison with ports. They want ba able to run most machines without
a compiling at all (BTW, our desktop users need the same), but setting
build farms when there are many machine roles is hard.
So packages need to be "equal in rights" in ports. The ports can have
things like this:
.if ${OSVERSION} < 700104 || ${OSVERSION} >= 900000
or this:
LIB_DEPENDS+= profiler.1:${PORTSDIR}/devel/google-perftools
but packages are not so flexible, all you have is:
@pkgdep perl-5.8.9_2
skv@ proposed the following changes:
* OPTIONS need radio-buttons (e.g. only one of MySQL, PostgreSQL,
SQLite) and dialog(1) supports it.
* Options must be included and installed to /var/db/ports/*/options
(this will allow to rebuild installed binary pkg as port)
* Info about options must be included to /var/db/pkg/*/+CONTENTS like:
@option WITH_SSL
@option WITHOUT_DEBUG
* Dependencies must be able to specify needed OPTIONS, both required to
set and required to be unset, somewaht like:
RUN_DEPENDS+= foobar:${PORTSDIR}/devel/foobar:+SSL:-SQLITE
This will allow to detect conflicts with installed packages with
incompatible options.
* For the package file names, introduce presets, e.g.:
OPTIONS_PRESETS= default "+SSL:-DEBUG" \
lite "-SSL:-DEBUG"
And preset name could be put to pkg name (may be "" for default).
* (internal) move away from CVS, rebalance to category-subcategory.
These ideas in later discussion evolved to another additions. Let say we
are able to use multiple repositories, where "repository" is a variant
of /usr/ports tree and packages built from it. Then, each port allows to
build several packages from it, with different options. Now, if we have
a port called "softina" and user does
pkg_add -r softina
then dependency search must be made given needed options. So @pkgdep
consists just of "softina" and versions like ">=1.1" and options. Also,
if packages are equal in rights to ports, they need integrity/security
check. So, package file name is now like:
softina-1.2.3_1:repo.id:1312929890.tbz # chars allowed by windows
Here is a repository id, just a hostname like "freebsd.org" for official
ports, and an unique build id in that repo, though it were suggested
that option preset name instead of that name will be better (because
human-readable). And `pkg_add -r' fetches a single file
softina.pkgmeta
consisting of sections for each actual package file. Each section has
a copy of what already is in .tbz file: name, comment, options,
dependencies (and their versions and options). And one thing not present
in .tbz - it's digital signature. Fetching pkg_add looks up local key
for given repository id to check. Fetching .pkgmeta beforehand also
allows to calculate if all needed dependencies are present in repo to
not fail in the middle of 100 packages as current pkg_add may do. The
signature is in another file and optional, so user could install the
plain .tbz file manually (it still contains all needed information,
.pkgmeta is only a copy except a signature).
The one other thing which is also optional to @pkgdep is again
repository id. This is to allow following situation:
* company has 10 it's own internal projects
* it also has 20 modified ports from original tree
* internal pkgs depend on modified ports, not original
* it's local port tree/pkg repo may hand only those, not full tree
Then internal pkgs may be depended on modified ports to be not
intermixed by mistake with original versions from official tree.
Repository IDs are used for that, this is optional mechanism, though.
End machines have two repositories in their config files, local and
offical, and all works correct, and local repo doesn't need to be a full
clone of ports tree.
The problem of -devel ports could be solved by using a global knob like
WITH_DEVEL (analogue of WITHOUT_X11), and options affect parts of
pkg-plist. The solved problem: glib has perl in R-deps, just for one
script, but this script is needed only for _build_ of dependent ports.
Now, if you install irssi, you will always have glib and perl, but you
don't need perl to run irssi. And irssi could have it's build dependence
of glib:+DEVEL and run of just plain glib.
Of course we don't want to split ports to perl, perl-base, perl-modules,
perl-doc, etc. like in Debian, and options could be solution here - one
port, several packages. Build farm may now build not one, but 2-3 common
option presets.
Here we go to another related problem, though - ports infrastructure.
I've read 16 chapters of http://www.netbsd.org/docs/pkgsrc/ and found
several interesting moments (let's concentrate on most close sibling of
our ports, though e.g. slots in Gentoo and Arch Linux system are also
worth looking too). They already have many of what we need, and since
pkg_install/ports by Jordan Hubbard is common ancestor, it may be easier
to port from them to us needed features.
For example, there is problem generating plist for our porters. And what
they have, a program to create port from distfile:
| Run the program url2pkg, which will ask you for a URL. Enter
| the URL of the distribution file (in most cases a .tar.gz file) and
| watch how the basic ingredients of your package are created
| automatically.
Just fix oddities later manually, but saves from all boilerplate work!
Next, skv@ said about radio-buttons, and thay also have it, in two
favors: first with one from group always set, second allowin all clear:
"Exactly one of the following gecko options is required"
"At most one of the following database options may be selected"
| PKG_OPTIONS_REQUIRED_GROUPS is like PKG_OPTIONS_OPTIONAL_GROUPS, but
| building the packages will fail if no option from the group is
| selected. PKG_OPTIONS_NONEMPTY_SETS is a list of names of sets of
| options. At least one option from each set must be selected.
The OPTIONS, however, are handled in different way:
$ grep "PKG.*OPTION" mk.conf
PKG_DEFAULT_OPTIONS= -arts -dvdread -esound
PKG_OPTIONS.kdebase= debug -sasl
PKG_OPTIONS.apache= suexec
Dependencies also allow versions:
BUILD_DEPENDS+= lua>=5.0:../../lang/lua
DEPENDS+= screen-[0-9]*:../../misc/screen
DEPENDS+= screen>=4.0:../../misc/screen
Checksum files for packages exist on their own, and only those may be
signed not the packages itself (an alternative to .pkgmeta approach
described above).
Another useful thing to borrow is patches/* separately from files/*
The most visible thing in pkgsrc superior to ports, is buildlink3
framework:
| Buildlink is a framework in pkgsrc that controls what headers and
| libraries are seen by a package's configure and build processes.
| [...] Please note that the normal system header and library paths,
| e.g. /usr/include, /usr/lib, etc., are always searched -- buildlink3
| is designed to insulate the package build from non-system-supplied
| software.
| [...]
| Some packages in pkgsrc install headers and libraries that coincide
| with headers and libraries present in the base system. Aside from
| a buildlink3.mk file, these packages should also include a
| builtin.mk file that includes the necessary checks to decide whether
| using the built-in software or the pkgsrc software is appropriate.
| [...] When building packages, it's possible to choose whether to set
| a global preference for using either the built-in (native) version
| or the pkgsrc version of software to satisfy a dependency. This is
| controlled by setting PREFER_PKGSRC and PREFER_NATIVE.
| PREFER_PKGSRC= yes
| PREFER_NATIVE= getopt skey tcp_wrappers
There are more automation:
| Up to now, the file PLIST, which contains a list of the files that
| are installed by the package, is nearly empty. Run
| bmake print-PLIST >PLIST
| to generate a probably correct list.
Yes, they rely on their derivative of BSD make, bmake, which it itself
worth merging deifferencies to our make (for example, there is a clean
and small alternative to autotools, mk-configure, using bmake and
written in style of .include <bsd.prog.mk>). There may be other examples
of userland tools worth porting from NetBSD to us instead of directly
upstream, e.g. awk... but that's another story, let's continue pkg.
Their developments of pkg_* tools contain facilities to implement a good
package manager out-of-the-box, e.g. pkg_add there has flags:
| -A Mark package as installed automatically, as dependency of
| another package. You can use
| pkg_admin set automatic=YES
| to mark packages this way after installation, and
| pkg_admin unset automatic
| to remove the mark. If you pkg_add a package without
| specifying -A after it had already been automatically
| installed, the mark is removed.
| -U Replace an already installed version from a package.
| Implies -u.
| -u If the package that's being installed is already installed,
| an update is performed.
These are crucial to effective binary package management.
> 4. Base system, closely related with packages.
The next in turn was concept of monolithic base system. The troubles
with base system were:
1. To change the components of the base you may only recompile it with
corresponding src.conf(5) options, but there is no track in base
system which are actually installed now. You cannot binary upgrade
a custom world, even if it is just unmodified -STABLE instead of
-RELEASE.
2. Consequently, there is no way to check integrity (MD5 etc.) of any
non-RELEASE variant (freebsd-update IDS is very limited).
3. No ties between base system and packages: who knows what previous
admin has installed, you may have compiler or may not have, etc.
Packages may silently broke if some part of base system SUDDENLY
disappears, as no dependency information is recorded.
4. Base system is monolithic, so there is no easy way to replace one
component with another - ports replacing base parts are hemorrhoids.
So, they conclude, the base system concept should be eliminated and be
split to bare kernel and gazillion of packages.
But we all know what benefits base system gives to us: it is actually
a *system* where all components are in concordance to each other, not
just a bunch of packages. Also, if it will be all split, this will
require *much more* efforts from our committers to keep everything in
sync. And our resources (efforts) are limited.
Actually, when you face two contradictory ways, all-or-nothing, both
with pros and cons - you should select neither of them, but try to find
something which will *synthesize* both of them. Such finding is an
interesting engineering (often inventor) task. And dougb@ already said
once that between base as-is and splitting all must be something middle.
It's Cathedral vs Bazaar, after all, and our Cathedral should be updated
and still use it's cathedral benefits. After all, NetBSD has proposal of
syspkg in 2004, and still has older format - just because it is not BSD
way, something new has to be invented.
The most obvious solution is to split base not to ~500 packages, but to,
say, ~50, and change boundaries. Why should freebsd.org developers place
boundaries in packages between themselves? So most natural solution is
to split packages out of base by the vendor criteria. Luckily, this work
is *already done* to some extent. So this will be minimal efforts
overhead, if any.
Looking at /usr/src/sontrib and http://wiki.freebsd.org/ContribSoftware
we can identify many of what could became a package. There can be
different approaches to criteria "what is in base system":
1. Only what is done on freebsd.org: all contrib must go to pkgs.
2. Whose effective vendor is now *@*bsd.org: contrib from other BSDs may
still live, and those with ceased upstream or renamed
(non-conflicting with ports) soft like libbsdxml, too.
3. Axe out only the most odious contrib parts: BIND (as Peter said in
archives, host/dig could be resurrected from Attic), sendmail (could
be replaced by dma) and several others, presumably GCC/binutils & CVS
(I've also heard about painful Kereberos interferencing with Samba).
The latter is most probable :-) given known conservatism and parts
inter-dependencies (openssl is required by freebsd-update, and it's not
*@*bsd.org - already an exception). But it still needs clear definition
of what base system is destined for.
A philosophical observation. The more "fat" base system is, the more
things are supposed about end user (the "WHEREs" in "SELECT"), so the
more narrow target audience is. The more thin base system is, the more
popular OS can be. You elitists may sleep well: FreeBSD will never be
as popular of Linux whose "rod" is just kernel, not base sys.
So each component of the base must be justified for the task, and for
task we may be should define to which user FreeBSD is targeted. Let it
be, say, task "the fresh setuped machine must be able to connect to
network to download and install binary packages". Then you don't need
GCC but you need editor to edit resolv.conf, less to view man pages,
tcpdump to debug network problems, send-pr to report a bug about
non-installed packages and thus a mail agent able to handle outgoing
report mail and daily periodic scripts to root while you have sex with
this (dma is suitable, sendmail is overkill). And so on.
Axing out GCC to packages has another benefit: the newer GCC could be
used, and base could be polished to be more compiler-agnostic (hello,
clang). But this requires binary packages to have equal rights with
ports, of course. A base without a compiler is not a dramatic - you
already need some ports to do "make release", and doc team already began
to split docs to packages, that's a right direction. For POLA reasons
all axed out packages (and sendmail too, respect traditions) should be
just packaged on CD1.
There may be another approach, not package one, but it is still in a fog
(and don't solve ports overriding base well). It's like the kernel and
modules: monolithic base system is just as monolithic kernel before
invention of modules - you the same way don't know what's in. Let's
dream a little - imagine we have our own DVCS, combining benefits of
centralized and distributed one, and not requiring splitting to multiple
repos like git, but allowing to use any given subset of files (not dirs
and subtrees as svn, *any* "inode"-based subset) from central repo
(this one "more equal" from all equal distributed repos). Such system
could be placed instead freebsd-update, binary updates of STABLE etc.
go to special branch, and user updates only those files which he have
on his system, VCS automatically tracks it.
> 5. Too short major releases' cycle.
I've once read a thread:
From: ***@FreeBSD.org
Subject: Schedule for releases
Date: Tue, 21 Dec 2010 09:47:08 -0800
where e.g. julian@ have said:
| Generally a company wouldn't want to go through the pain of an OS
| upgrade more than about once in three years and often it's longer..
| It IS a pain for them.
And many business people reported the same. And it is known many people
are doing backports, testing etc., they should be motivated to give work
back to FreeBSD. While it is more social problem, it is also a
DVCS/bugtracker problem, but more rare major releases would still help
on it's own.
It is known why the current scheme was adopted: the 4.11/5.3 case,
a horror. But between X.4 and X.11 there are even *several* intermediate
choices. What happens today: many conservative users (or builders of
products) consider -STABLE is really stable at the X.2 release. But just
after than an (X+1).0 is forked and all developers' attention goes to
new branch, X.3 and X.4 are not seriously developed. And due to
timelines described above, many users will then upgrade right away to
(X+2).Y, not (X+1). There are many 6.x and 7.x users in the wild, many
of 7.x users will upgrade directly to 9.x skipping 8.x. Is this good?
No.
Aside of many branches receive not enought production testing, our
committers must do MFCs to TWO branches, stable/7 and stable/8. While
usualy having no real possibility to test properly. Nonsense.
Is supporting two stable branches not using more efforts that it was in
4.x times? Not so? Still could be made better.
Proposed solution: prolonging major releases fork time a little. Just to
time so only one stable branch will exist. I hope increasing branch
lifetime to one minor release will help: last will be not .4, but .5,
and new .0 fork should occur at least after .3, not .2. We are not
Ubuntu. I understand that pace of changes is high, 3 years for each .0
may be unacceptable, but waht about at least 2.5 years? Not like 1.8
years currently. Rememeber, .5 and even .6 is still far from .11.
> 6. Bug tracker, unicode and other less important trivia.
GNATS is too old and unfriendly e.g. to user attachments. It has only
one adavantage: user is able to send report from CLI by mail without any
registration. This is essential. May be the good alternative will be RT
used at cpan.org (it also accepts by mail). Hopefully this will help
a little to solving problems. Not sure, though - and unsolved bugs are
also make users to go away from FreeBSD.
Users also want properly working UTF-8 out-of-the-box. Not only console,
but full collation support, etc.
There was suggestion about automatic kldloading of drivers, but this
work already began (for USB) in June - more matte of our documentation
and press relations, huh.
Installer. Must be more featureful and more user friendly :-) This is
often may give negative first impression if installer was unable to do
something even if system after this could work well.
Other problems, like broken cvsup, are exist, but are not critical to
Project's survival.
> FreeBSD strengths to be concentrated on.
The paragraph about virtualization above, as long as points below, are
roughly translated words of one small business representative in Russia,
who often acts as integrator.
1. BSD License. Very good for embedded vendors, we should be able to
compile and work both base and ports without licenses requiring to
open the sources. It would be good to not loose functionality without
them.
2. Kernel features for storage (ZFS, HAST, GEOM). We need to go to
NAS/SAN solutions niche while we have ability - it is still, because
Solaris etc. in unknown state with Oracle, BTRFS still beta. We have
about 1 year max. It would be nice to roll out "box" solution for a
failover iSCSI storage (2 PCs with ZFS raids replicated by HAST and
accessible by iSCSI with automatic role change).
3. Kernel features for complex network solutions (netgraph, carp, ipfw).
The niche for routers & traffic analysis is still ours. It would be
nice to take e.g. pfSense and agree with some vendor (Netgear,
D-Link, etc) to put on sale hardware with FreeBSD inside.
So these are main ways - embedded (NAS & routers). Other market is still
not our, e.g. not an application server until there will be a killer-app
for SCTP. May be also for a DB: Solaris was first recommended for
PostgreSQL, with some efforts we may become the first. Of course, the
main way - if some commercial company will offer FreeBSD.
We still have some time, but almost no time. We need to take decisions
right now.
That's all for today. Thanks to everyone who has patience to carefully
read this all entirely.
--
WBR, Vadim Goncharov. ICQ#166852181 mailto:***@mail.ru
[Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight]
I have bad news.
A month ago techlead of Rambler's Mail department declared in his blog
that they begin migration from FreeBSD to Debian/Ubuntu. Comments to the
blog entry (there was much flame) revealed that search department of
Russian search engine #1 Yandex.com also plans to migrate their
search cluster - about 30,000 servers in several DCs (~60% of total
their servers - to Linux. The problem here is that both big companies
were using FreeBSD from the beginning (~1997), so migration will be
rather expensive.
The official reasons (really semi-official, as these are individual
blogs), in short, were:
* inadequate package manager and huge monolithic base system
* lack of OpenVZ-like virtualization (need CPU limiting)
* a FreeBSD marketshare forecast for 5 years
Despite of several our committers working for them, the operating
expenses for FreeBSD are considered too high by these companies. They've
not confirmed, but the key problem is probably shortage of FreeBSD
specialists on the market to hire (e.g. Yandex has about 20 FreeBSD
admins and about 84 Linux admins, for all other their services and
40% of servers). So this is problem of FreeBSD's too low
userbase/marketshare, caused, in turn, by other FreeBSD's problems.
Given that they is just planning today, but not yet started, we have to
solve our problems or FreeBSD will effectively die[1]. Currently
https://ssl.netcraft.com/ssl-sample-report//CMatch/oscnt_all shows that
still 3% of servers are running FreeBSD. We are currently probably at
the tip of 'hype curve'[2] and then our userbase will begin to shrink,
and the task is to solve problems, so after hard time it will rise again
(hopefully more than it was before).
[1] 'Die' means shrinking userbase size to those of NetBSD/OpenBSD.
[2] http://www.metodolog.ru/01493/2.gif
Earlier this year in this list marcel@ have said about objective view:
http://lists.freebsd.org/pipermail/freebsd-arch/2011-January/010999.html
from the outside, not FreeBSD people. I have posted a call to Russian
community in my blog (http://nuclight.livejournal.com/128712.html) and
gathered a list of problems along with possible solutions to some of
them. The rest of this message is summary of them.
In general, people sympathizing FreeBSD agree that there are no fatal
technical problems and that FreeBSD could compete with mainstream Linux.
That said, the 80/20 principle is in effect and 80% losed users for
FreeBSD are caused by 20% problems. The trouble here is that what
"outside" people view as considerable problems, we here inside FreeBSD
view as something insignificant ("do it yourself"). Thus 80% could be
achieved by 20% of work. I have sorted them in order of decreased
importance (priority):
1. Social problems of community (and marketing, docs, ...)
2. Lack of drivers and virtualization.
3. Ports and packages.
4. Base system, closely related with packages.
5. Too short major releases' cycle.
6. Bug tracker, unicode and other less important trivia.
(bonus) FreeBSD strengths to be concentrated on.
Now all of them more detailed.
> 1. Social (psychologic) problems of community (marketing, docs, ...).
This is the most important one, because all technical problems are just
won't get solved because are even not viewed as problems. The FreeBSD
Project does not listen to users' needs. The typical response when poor
user want something is: "we don't need this, we won't change for you",
with "where are your patches?" at best. Then many users go out when see
such attitude toward them.
The key points are:
1) *The competent user is not zealot*.
2) The system is *for users, not for developers*.
With first: when user sees the system costs too much time for him, and
that those problems won't get fixed and even considered such - he will
switch to another OS. This may mean that he is able to follow our advice
hoe to do some or another thing (e.g. to recompile all ports), but this
is unacceptable to him because this is too much maintenance (another
system by objectivity requires less work). Many businesses fall into
this category. They will not with FreeBSD by any price just because they
love OS. Unlike ourselves.
With second: that's simple and not simple. It is simple in that by
nature each person don't make a work just for himself but for all
others, who are now "users" to him, too - just by induction that's not
for developers only. It is not simple due to question "Why do we need
those who don't contribute anything back to us?" which random committer
has right to ask. The answer, in short, is: there tasks which could be
only done by large groups of people, e.g. big corporations. For
corporations to invest and contribute back - the system must be popular
enough. To be popular enough means there must be a big amount of users
who is not contributing anything. It is useful in itself, though, that
some of those users, say 1%, will become contributors, that is, absolute
number of our developers will also benefit from FreeBSD being much more
popular.
There are opinions in our community like:
| > Personally I'd like to think that that we write an OS for users.
|
| We write an OS for the people who can and will use an OS written by
| us.
| FreeBSD is whatever its developers make it be
These are wrong and harmful nowadays. The world has changed. We have to
admit it or we will die. We have to admit mistakes and *change* some of
the ways we're doing things: any answer like "I don't agree, we don't
need these users/features/etc., we *won't* do this for someone" - is
just another step to grave.
Of course that doesn't mean to do everything - we often have not enough
time/resources to do. But that's another question and should not be
mixed with attitude - "it's good but we have no resources, will you do?"
vs "we don't need this, go away" (that's not only about features but
about keeping something too).
So ("system is for users") we need to (re)define who our user is. I view
the current situation as following:
100% .---------------. The stratification of overall
| Ubuntu | our userbase. Areas are:
| target | 6
| audience | 1 - official developers
expand -> |---------------|
to here | not our, but | 2 - actively participating and
| competent | 5 sending patches, contributors
| users | and other "zealots" (whose
|===============| "soul is with FreeBSD")
|announce@ only | 4
involved |---------------| 3 - not so active but discussing
| subscribed to | in mail lists, patches go
| our mailing | 3 sometimes
| lists |
committed |- - - - - - - -| 4 - busy with work to read lists
| | 2
|_______________| 5 - can use, but costs are now high
| *@freebsd.org | 1
0% `---------------' 6 - we'll never want them
The terms "involved" and "committed" were taken from
http://en.wikipedia.org/wiki/The_Chicken_and_the_Pig for those who are
"at the heart" of the community, at "periphery" and all others.
Currently, only those in (1) and partially (2) are listened to,
sometimes with "call for ..." in (3). Meanwhile, the most important of
our users, who are using FreeBSD in production, who are busy with real
work in real world, are in (4) and sometimes (5). Those are largely
ignored by the Project - even messages to announce@ last years are less
about important events and calls for volunteers/money.
The other social problem is lack of companies which offer commercial
support of FreeBSD like RedHat does.
As a possible solution to find out what user needs are, and to
compensate the fact that not all users in (4) and (5) care enough about
our principles and spirit, I propose a system for "weighted democracy".
This is a voting system (I could implement it), say, there is a mail
***@freebsd.org to which users send filled in vote forms, with
selected answers from a survey published in ***@.
The system has a database where users are recorded given their FreeBSD
activity in mail lists etc., and votes are summed as follows:
+ 1.00 vote for anyone not in database (not involved to FreeBSD)
+ a decimal logarithm of number of posts to mail lists (2 votes for 100
posts, 3 votes for 1000 posts)
+ a binary logarithm of num PRs in GNATS db (6 votes for 64 send-pr's)
+ a proportion (say, N/2) of entries for this person in "svn log" (e.g.
in "Submitted by:")
+ an assigned (by core@) number of votes in special exceptional DB (for
corporation and the like)
The system then presents results for each answer: 1) how many users
voted, 2) how many votes summed.
This is by no mean to measure "exact contribution", but to defend from
anonymous users and trolls who not care about amount of work developers
will need to. The results are viewed as a feedback to core team, not as
a final decision - that's all for what marketing is solely exist. There
no adequate feedback from users to developers currently at all -
individual posts in mail lists are too small for statistics (what about
ministat(1) for users, huh?..).
What is also in this part? Documentation. No, not in that sense. The
observations show that while there are described solutions to problems
in system, it's difficult for users to find it. It's somehow organized
or misorganized that solution is not intuitive for them (e.g. someone
migrating from Windows complained that it was difficult to find 'simply
how to format a flash drive' in Handbook). I don't know how to handle
this properly. May be catalogues with Best Practices howto's pointing to
exact sections of Handbook, FAQ, articles etc.? May be better search
mechanisms?..
And the last here about too much conservatism and rigidness in our camp.
As one of the opponents said (roughly translated):
| FreeBSD, historically, was good system, let's say, "valid system
| for solid people". Pureness, strictness, etc. The base system is
| consequence of this approach. The trouble is, that in 2011 nobody
| needs pureness and strictness. Customers need transparency and fast
| reaction to their requests. If there is no transparency and speed,
| you get CentOS 6 (just released, at last). Or FreeBSD (Gentoo,
| Solaris, you name it).
I don't agree that those certainly contradicts each other, but we
definitely need to change, er, something. And don't keep something just
because it is tradition, if that tradition has no more technical reasons
for our users. Let's shift conservatism to another field where it is
really needed (e.g. legacy support and other examples below).
Finally, two more quotes from the ***@freebsd.org archives of last
2 years:
| From: Cyrille Lefevre <cyrille.lefevre-***@laposte.net>
| [...] however, you answer remember me why I quit FreeBSD, cease
| fire, courtesy is the key
and
| From: Julian H. Stacey
| Though I & many others have sent lots of send-pr, contributing even
| a spelling correction to FreeBSD is much harder than to e.g.
| http://wikipedia.org
The threshold of entering to FreeBSD is too high and should be lowered.
I can confirm the last quote on my own example, 2 months ago. I've
submitted a patch to ipfw, adding call/return rule actions. It allows
to organize rules in somewhat like pf anchors, and, more importantly,
iptables chains, enabling FreeBSD ipfw to compete with Linux at least
partially in this sphere. The patch wasn't taken in maillists, I had
to push it in IRC, and push hard: the whole needed by many big users
thing was deferred (and still not MFCed to 8.x) just because some
#define's and printf's were Not So Proper & Right Way! F*cking shame.
> 2. Lack of drivers and poor guest virtualization.
It is known that FreeBSD supports less hardware than competitors. It is,
however, a matter of popularity - the more system will have users, the
more drivers will be created by 3rd parties, so we cannot directly
affect this. It's kind of exclusive circle.
But there is one area - virtualization - without supporting it the
FreeBSD niche will collapse very fast. It is necessary to say that it is
about guest (para-)virtualization only, not host mode. Host mode market
is already lost for FreeBSD for quite some time, may be something in the
future, but today it will waste of resources. What is needed here is
just analogue to OpenVZ (and jails/rctl already done more than half of
the work here).
And in guest mode FreeBSD *urgently needs* working drivers/utilities for
all common suits, Citrix XenServer, VMWare vSphere (ESXi), Xen*,
Microsoft Hyper-V, etc. This must be the primary direction for
developers' forces and FreeBSD Foundation's money. We have may be about
1 year here. Why? Because of cloud computing and VPS/VDS. It's no
matter what OS runs hoster if clients will use FreeBSD and we still have
users. I've been already asked by a customer for which I wrote
a non-portable kqueue-based daemon to rewrite for Linux because they
want to go for Amazon EC2 (it's cheaper as only used resources are
accounted) and FreeBSD there has still many in ISSUES scaring them...
> 3. Ports and packages.
What was the main problems with large-scale installations of FreeBSD in
that businesses? In short, that binary packages are not equal in rights
to ports, and that complicates things (i.e. requires too much work) when
one have many (> 10) servers. This was listed to me as:
1) No pkg and pkg-devel versions. The -devel version is headers, static
libs, programmer examples, etc. not needed in production (we could
say this part is what is actually depended on in B-deps).
2) Package name is dependent on options, so packages with another opts
don't work well when dependencies are rebuilt.
3) Conflicts: no way to have apache13 and apache22 the same time.
4) No dependence on base system. You may cut out something, recompile
world, deploy it on cluster and just then see that some packages are
now don't work.
5) Dependencies are badly designed. No version ranges in dependencies,
no alternative packages, no priorities in package search.
6) Update problems. The version is just coded into name of package, and
dependencies are on the entire name, so there are situations when
install/upgrade of just one package may require rebuild 3/4 of all
pkgs. You cannot easiy modify installed package without editing pkgdb
manually. It is impossible to upgrade/replace package by out of the
box tools.
7) Base system has no "out of the box" tools for package upgrade. Our
business opponents say this the least problem as one can always
install portupgrade, but conclude that overall base system concept
does not play well with full-featured packages (see also next part
about base system).
8) There is no -STABLE supported branches in ports.
All of this could be avoided (they know about tinderbox etc.) but just
requires too much work, for their basic tasks like automated upgrade of
entire system & packages or reinstall of needed packages.
That's problems. Next, possible (gathered) solutions.
It is obvious that current packages are not first-class citizens, in
comparison with ports. They want ba able to run most machines without
a compiling at all (BTW, our desktop users need the same), but setting
build farms when there are many machine roles is hard.
So packages need to be "equal in rights" in ports. The ports can have
things like this:
.if ${OSVERSION} < 700104 || ${OSVERSION} >= 900000
or this:
LIB_DEPENDS+= profiler.1:${PORTSDIR}/devel/google-perftools
but packages are not so flexible, all you have is:
@pkgdep perl-5.8.9_2
skv@ proposed the following changes:
* OPTIONS need radio-buttons (e.g. only one of MySQL, PostgreSQL,
SQLite) and dialog(1) supports it.
* Options must be included and installed to /var/db/ports/*/options
(this will allow to rebuild installed binary pkg as port)
* Info about options must be included to /var/db/pkg/*/+CONTENTS like:
@option WITH_SSL
@option WITHOUT_DEBUG
* Dependencies must be able to specify needed OPTIONS, both required to
set and required to be unset, somewaht like:
RUN_DEPENDS+= foobar:${PORTSDIR}/devel/foobar:+SSL:-SQLITE
This will allow to detect conflicts with installed packages with
incompatible options.
* For the package file names, introduce presets, e.g.:
OPTIONS_PRESETS= default "+SSL:-DEBUG" \
lite "-SSL:-DEBUG"
And preset name could be put to pkg name (may be "" for default).
* (internal) move away from CVS, rebalance to category-subcategory.
These ideas in later discussion evolved to another additions. Let say we
are able to use multiple repositories, where "repository" is a variant
of /usr/ports tree and packages built from it. Then, each port allows to
build several packages from it, with different options. Now, if we have
a port called "softina" and user does
pkg_add -r softina
then dependency search must be made given needed options. So @pkgdep
consists just of "softina" and versions like ">=1.1" and options. Also,
if packages are equal in rights to ports, they need integrity/security
check. So, package file name is now like:
softina-1.2.3_1:repo.id:1312929890.tbz # chars allowed by windows
Here is a repository id, just a hostname like "freebsd.org" for official
ports, and an unique build id in that repo, though it were suggested
that option preset name instead of that name will be better (because
human-readable). And `pkg_add -r' fetches a single file
softina.pkgmeta
consisting of sections for each actual package file. Each section has
a copy of what already is in .tbz file: name, comment, options,
dependencies (and their versions and options). And one thing not present
in .tbz - it's digital signature. Fetching pkg_add looks up local key
for given repository id to check. Fetching .pkgmeta beforehand also
allows to calculate if all needed dependencies are present in repo to
not fail in the middle of 100 packages as current pkg_add may do. The
signature is in another file and optional, so user could install the
plain .tbz file manually (it still contains all needed information,
.pkgmeta is only a copy except a signature).
The one other thing which is also optional to @pkgdep is again
repository id. This is to allow following situation:
* company has 10 it's own internal projects
* it also has 20 modified ports from original tree
* internal pkgs depend on modified ports, not original
* it's local port tree/pkg repo may hand only those, not full tree
Then internal pkgs may be depended on modified ports to be not
intermixed by mistake with original versions from official tree.
Repository IDs are used for that, this is optional mechanism, though.
End machines have two repositories in their config files, local and
offical, and all works correct, and local repo doesn't need to be a full
clone of ports tree.
The problem of -devel ports could be solved by using a global knob like
WITH_DEVEL (analogue of WITHOUT_X11), and options affect parts of
pkg-plist. The solved problem: glib has perl in R-deps, just for one
script, but this script is needed only for _build_ of dependent ports.
Now, if you install irssi, you will always have glib and perl, but you
don't need perl to run irssi. And irssi could have it's build dependence
of glib:+DEVEL and run of just plain glib.
Of course we don't want to split ports to perl, perl-base, perl-modules,
perl-doc, etc. like in Debian, and options could be solution here - one
port, several packages. Build farm may now build not one, but 2-3 common
option presets.
Here we go to another related problem, though - ports infrastructure.
I've read 16 chapters of http://www.netbsd.org/docs/pkgsrc/ and found
several interesting moments (let's concentrate on most close sibling of
our ports, though e.g. slots in Gentoo and Arch Linux system are also
worth looking too). They already have many of what we need, and since
pkg_install/ports by Jordan Hubbard is common ancestor, it may be easier
to port from them to us needed features.
For example, there is problem generating plist for our porters. And what
they have, a program to create port from distfile:
| Run the program url2pkg, which will ask you for a URL. Enter
| the URL of the distribution file (in most cases a .tar.gz file) and
| watch how the basic ingredients of your package are created
| automatically.
Just fix oddities later manually, but saves from all boilerplate work!
Next, skv@ said about radio-buttons, and thay also have it, in two
favors: first with one from group always set, second allowin all clear:
"Exactly one of the following gecko options is required"
"At most one of the following database options may be selected"
| PKG_OPTIONS_REQUIRED_GROUPS is like PKG_OPTIONS_OPTIONAL_GROUPS, but
| building the packages will fail if no option from the group is
| selected. PKG_OPTIONS_NONEMPTY_SETS is a list of names of sets of
| options. At least one option from each set must be selected.
The OPTIONS, however, are handled in different way:
$ grep "PKG.*OPTION" mk.conf
PKG_DEFAULT_OPTIONS= -arts -dvdread -esound
PKG_OPTIONS.kdebase= debug -sasl
PKG_OPTIONS.apache= suexec
Dependencies also allow versions:
BUILD_DEPENDS+= lua>=5.0:../../lang/lua
DEPENDS+= screen-[0-9]*:../../misc/screen
DEPENDS+= screen>=4.0:../../misc/screen
Checksum files for packages exist on their own, and only those may be
signed not the packages itself (an alternative to .pkgmeta approach
described above).
Another useful thing to borrow is patches/* separately from files/*
The most visible thing in pkgsrc superior to ports, is buildlink3
framework:
| Buildlink is a framework in pkgsrc that controls what headers and
| libraries are seen by a package's configure and build processes.
| [...] Please note that the normal system header and library paths,
| e.g. /usr/include, /usr/lib, etc., are always searched -- buildlink3
| is designed to insulate the package build from non-system-supplied
| software.
| [...]
| Some packages in pkgsrc install headers and libraries that coincide
| with headers and libraries present in the base system. Aside from
| a buildlink3.mk file, these packages should also include a
| builtin.mk file that includes the necessary checks to decide whether
| using the built-in software or the pkgsrc software is appropriate.
| [...] When building packages, it's possible to choose whether to set
| a global preference for using either the built-in (native) version
| or the pkgsrc version of software to satisfy a dependency. This is
| controlled by setting PREFER_PKGSRC and PREFER_NATIVE.
| PREFER_PKGSRC= yes
| PREFER_NATIVE= getopt skey tcp_wrappers
There are more automation:
| Up to now, the file PLIST, which contains a list of the files that
| are installed by the package, is nearly empty. Run
| bmake print-PLIST >PLIST
| to generate a probably correct list.
Yes, they rely on their derivative of BSD make, bmake, which it itself
worth merging deifferencies to our make (for example, there is a clean
and small alternative to autotools, mk-configure, using bmake and
written in style of .include <bsd.prog.mk>). There may be other examples
of userland tools worth porting from NetBSD to us instead of directly
upstream, e.g. awk... but that's another story, let's continue pkg.
Their developments of pkg_* tools contain facilities to implement a good
package manager out-of-the-box, e.g. pkg_add there has flags:
| -A Mark package as installed automatically, as dependency of
| another package. You can use
| pkg_admin set automatic=YES
| to mark packages this way after installation, and
| pkg_admin unset automatic
| to remove the mark. If you pkg_add a package without
| specifying -A after it had already been automatically
| installed, the mark is removed.
| -U Replace an already installed version from a package.
| Implies -u.
| -u If the package that's being installed is already installed,
| an update is performed.
These are crucial to effective binary package management.
> 4. Base system, closely related with packages.
The next in turn was concept of monolithic base system. The troubles
with base system were:
1. To change the components of the base you may only recompile it with
corresponding src.conf(5) options, but there is no track in base
system which are actually installed now. You cannot binary upgrade
a custom world, even if it is just unmodified -STABLE instead of
-RELEASE.
2. Consequently, there is no way to check integrity (MD5 etc.) of any
non-RELEASE variant (freebsd-update IDS is very limited).
3. No ties between base system and packages: who knows what previous
admin has installed, you may have compiler or may not have, etc.
Packages may silently broke if some part of base system SUDDENLY
disappears, as no dependency information is recorded.
4. Base system is monolithic, so there is no easy way to replace one
component with another - ports replacing base parts are hemorrhoids.
So, they conclude, the base system concept should be eliminated and be
split to bare kernel and gazillion of packages.
But we all know what benefits base system gives to us: it is actually
a *system* where all components are in concordance to each other, not
just a bunch of packages. Also, if it will be all split, this will
require *much more* efforts from our committers to keep everything in
sync. And our resources (efforts) are limited.
Actually, when you face two contradictory ways, all-or-nothing, both
with pros and cons - you should select neither of them, but try to find
something which will *synthesize* both of them. Such finding is an
interesting engineering (often inventor) task. And dougb@ already said
once that between base as-is and splitting all must be something middle.
It's Cathedral vs Bazaar, after all, and our Cathedral should be updated
and still use it's cathedral benefits. After all, NetBSD has proposal of
syspkg in 2004, and still has older format - just because it is not BSD
way, something new has to be invented.
The most obvious solution is to split base not to ~500 packages, but to,
say, ~50, and change boundaries. Why should freebsd.org developers place
boundaries in packages between themselves? So most natural solution is
to split packages out of base by the vendor criteria. Luckily, this work
is *already done* to some extent. So this will be minimal efforts
overhead, if any.
Looking at /usr/src/sontrib and http://wiki.freebsd.org/ContribSoftware
we can identify many of what could became a package. There can be
different approaches to criteria "what is in base system":
1. Only what is done on freebsd.org: all contrib must go to pkgs.
2. Whose effective vendor is now *@*bsd.org: contrib from other BSDs may
still live, and those with ceased upstream or renamed
(non-conflicting with ports) soft like libbsdxml, too.
3. Axe out only the most odious contrib parts: BIND (as Peter said in
archives, host/dig could be resurrected from Attic), sendmail (could
be replaced by dma) and several others, presumably GCC/binutils & CVS
(I've also heard about painful Kereberos interferencing with Samba).
The latter is most probable :-) given known conservatism and parts
inter-dependencies (openssl is required by freebsd-update, and it's not
*@*bsd.org - already an exception). But it still needs clear definition
of what base system is destined for.
A philosophical observation. The more "fat" base system is, the more
things are supposed about end user (the "WHEREs" in "SELECT"), so the
more narrow target audience is. The more thin base system is, the more
popular OS can be. You elitists may sleep well: FreeBSD will never be
as popular of Linux whose "rod" is just kernel, not base sys.
So each component of the base must be justified for the task, and for
task we may be should define to which user FreeBSD is targeted. Let it
be, say, task "the fresh setuped machine must be able to connect to
network to download and install binary packages". Then you don't need
GCC but you need editor to edit resolv.conf, less to view man pages,
tcpdump to debug network problems, send-pr to report a bug about
non-installed packages and thus a mail agent able to handle outgoing
report mail and daily periodic scripts to root while you have sex with
this (dma is suitable, sendmail is overkill). And so on.
Axing out GCC to packages has another benefit: the newer GCC could be
used, and base could be polished to be more compiler-agnostic (hello,
clang). But this requires binary packages to have equal rights with
ports, of course. A base without a compiler is not a dramatic - you
already need some ports to do "make release", and doc team already began
to split docs to packages, that's a right direction. For POLA reasons
all axed out packages (and sendmail too, respect traditions) should be
just packaged on CD1.
There may be another approach, not package one, but it is still in a fog
(and don't solve ports overriding base well). It's like the kernel and
modules: monolithic base system is just as monolithic kernel before
invention of modules - you the same way don't know what's in. Let's
dream a little - imagine we have our own DVCS, combining benefits of
centralized and distributed one, and not requiring splitting to multiple
repos like git, but allowing to use any given subset of files (not dirs
and subtrees as svn, *any* "inode"-based subset) from central repo
(this one "more equal" from all equal distributed repos). Such system
could be placed instead freebsd-update, binary updates of STABLE etc.
go to special branch, and user updates only those files which he have
on his system, VCS automatically tracks it.
> 5. Too short major releases' cycle.
I've once read a thread:
From: ***@FreeBSD.org
Subject: Schedule for releases
Date: Tue, 21 Dec 2010 09:47:08 -0800
where e.g. julian@ have said:
| Generally a company wouldn't want to go through the pain of an OS
| upgrade more than about once in three years and often it's longer..
| It IS a pain for them.
And many business people reported the same. And it is known many people
are doing backports, testing etc., they should be motivated to give work
back to FreeBSD. While it is more social problem, it is also a
DVCS/bugtracker problem, but more rare major releases would still help
on it's own.
It is known why the current scheme was adopted: the 4.11/5.3 case,
a horror. But between X.4 and X.11 there are even *several* intermediate
choices. What happens today: many conservative users (or builders of
products) consider -STABLE is really stable at the X.2 release. But just
after than an (X+1).0 is forked and all developers' attention goes to
new branch, X.3 and X.4 are not seriously developed. And due to
timelines described above, many users will then upgrade right away to
(X+2).Y, not (X+1). There are many 6.x and 7.x users in the wild, many
of 7.x users will upgrade directly to 9.x skipping 8.x. Is this good?
No.
Aside of many branches receive not enought production testing, our
committers must do MFCs to TWO branches, stable/7 and stable/8. While
usualy having no real possibility to test properly. Nonsense.
Is supporting two stable branches not using more efforts that it was in
4.x times? Not so? Still could be made better.
Proposed solution: prolonging major releases fork time a little. Just to
time so only one stable branch will exist. I hope increasing branch
lifetime to one minor release will help: last will be not .4, but .5,
and new .0 fork should occur at least after .3, not .2. We are not
Ubuntu. I understand that pace of changes is high, 3 years for each .0
may be unacceptable, but waht about at least 2.5 years? Not like 1.8
years currently. Rememeber, .5 and even .6 is still far from .11.
> 6. Bug tracker, unicode and other less important trivia.
GNATS is too old and unfriendly e.g. to user attachments. It has only
one adavantage: user is able to send report from CLI by mail without any
registration. This is essential. May be the good alternative will be RT
used at cpan.org (it also accepts by mail). Hopefully this will help
a little to solving problems. Not sure, though - and unsolved bugs are
also make users to go away from FreeBSD.
Users also want properly working UTF-8 out-of-the-box. Not only console,
but full collation support, etc.
There was suggestion about automatic kldloading of drivers, but this
work already began (for USB) in June - more matte of our documentation
and press relations, huh.
Installer. Must be more featureful and more user friendly :-) This is
often may give negative first impression if installer was unable to do
something even if system after this could work well.
Other problems, like broken cvsup, are exist, but are not critical to
Project's survival.
> FreeBSD strengths to be concentrated on.
The paragraph about virtualization above, as long as points below, are
roughly translated words of one small business representative in Russia,
who often acts as integrator.
1. BSD License. Very good for embedded vendors, we should be able to
compile and work both base and ports without licenses requiring to
open the sources. It would be good to not loose functionality without
them.
2. Kernel features for storage (ZFS, HAST, GEOM). We need to go to
NAS/SAN solutions niche while we have ability - it is still, because
Solaris etc. in unknown state with Oracle, BTRFS still beta. We have
about 1 year max. It would be nice to roll out "box" solution for a
failover iSCSI storage (2 PCs with ZFS raids replicated by HAST and
accessible by iSCSI with automatic role change).
3. Kernel features for complex network solutions (netgraph, carp, ipfw).
The niche for routers & traffic analysis is still ours. It would be
nice to take e.g. pfSense and agree with some vendor (Netgear,
D-Link, etc) to put on sale hardware with FreeBSD inside.
So these are main ways - embedded (NAS & routers). Other market is still
not our, e.g. not an application server until there will be a killer-app
for SCTP. May be also for a DB: Solaris was first recommended for
PostgreSQL, with some efforts we may become the first. Of course, the
main way - if some commercial company will offer FreeBSD.
We still have some time, but almost no time. We need to take decisions
right now.
That's all for today. Thanks to everyone who has patience to carefully
read this all entirely.
--
WBR, Vadim Goncharov. ICQ#166852181 mailto:***@mail.ru
[Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight]