tags/debianzack's home pagehttp://upsilon.cc/~zack/tags/debian/zack's home pageikiwiki2016-11-28T10:27:43Zlast week to take part in the Debian Contributors Surveyhttp://upsilon.cc/~zack/blog/posts/2016/11/last_week_to_take_part_in_the_Debian_Contributors_Survey/2016-11-28T10:27:43Z2016-11-28T10:27:43Z
<h1>Debian Contributors Survey 2016</h1>
<p>About 3 weeks ago, together with Molly and Mathieu, we launched
the first edition of the Debian Contributors Survey. I won't harp
on it any further, because you can find all relevant information
about it <a href=
"https://bits.debian.org/2016/11/debian-contributors-survey-2016.html">
on the Debian blog</a> or as part of the <a href=
"https://lists.debian.org/debian-devel-announce/2016/11/msg00003.html">
original announcement</a>.</p>
<p>But it's worth noting that you've now only <strong>one week left
to participate</strong> if you want to: the deadline for
participation is <strong>4 December 2016, at 23:59
UTC</strong>.</p>
<p>If you're a Debian contributor and would like to participate,
just go to the <a href=
"http://debian.limequery.org/696747"><strong>survey participation
page</strong></a> and fill in!</p>
Shuttleworth Foundation Flash Granthttp://upsilon.cc/~zack/blog/posts/2014/12/Shuttleworth_Foundation_Flash_Grant/2014-12-05T15:56:18Z2014-12-05T15:51:19Z
<h1>Shuttleworth Foundation Flash Grant</h1>
<p>I'm glad to announce that I've been awarded a 5,000 USD "Flash
Grant" by the <a href=
"https://www.shuttleworthfoundation.org/">Shuttleworth
Foundation</a>.</p>
<p>Flash grants are an interesting funding model, which I've just
learned about. You don't need to apply for them. Rather, you get
nominated by current <a href=
"https://www.shuttleworthfoundation.org/fellows/">fellows</a>, and
then selected and approached by the foundation for funding. The
grant amount is smaller than actual <a href=
"https://www.shuttleworthfoundation.org/applications/">fellowships</a>,
but it comes with very few strings attached: furthering open
knowledge (which is the foundation's <a href=
"https://www.shuttleworthfoundation.org/about/">core mission</a>)
and being transparent about how you use the money.</p>
<p>I'm lucky enough to already have a full-time job to pay my
bills, and I do my Free Software activism mostly in my spare time.
So I plan to use the money not to pay my bills, but rather to boost
the parts of my Free Software activities that could benefit from
some funding. I don't have a fully detailed budget yet but,
tentatively: some money will go to fund <a href=
"http://sources.debian.net">Debsources</a> development (by others),
some into promoting my <a href=
"http://meetings-archive.debian.net/pub/debian-meetings/2014/debconf14/webm/Debian_in_the_Dark_Ages_of_Free_Software.webm">
thoughts on the dark ages of Free Software</a>, and maybe some into
helping the upcoming <a href=
"https://www.debian.org/releases/jessie/">release</a> of Debian.
I'll provide a public report at the end of the funding period (~6
months from now).</p>
<p>I'd like to thank the Shuttleworth Foundation for the grant and
foundation's fellow <a href="http://jonasoberg.net/">Jonas
Öberg</a> for making this possible.</p>
CTTE nominationhttp://upsilon.cc/~zack/blog/posts/2014/11/CTTE_nomination/2014-11-27T20:43:01Z2014-11-27T20:43:01Z
<p>Apparently, enough fellow developers have been foolish enough to
<a href=
"https://lists.debian.org/debian-devel-announce/2014/11/msg00013.html">
nominate</a> me as a prospective member of the <a href=
"https://www.debian.org/devel/tech-ctte">Debian Technical
Committee</a> (CTTE), that I've been approached to formally
accept/decline the nomination. <small>(Accepted nominees would then
go through a selection process and possibly proposed to the DPL for
nomination.)</small></p>
<p>I'm honored by the nominations and I thank the fellow developers
that have thrown my name in the hat. But I've respectfully declined
the nomination. Given my current involvement in an ongoing <a href=
"https://lists.debian.org/debian-vote/2014/11/msg00274.html">attempt</a>
to introduce a maximum term limit for CTTE membership, it would
have been highly inappropriate for me to accept the nomination at
this time.</p>
<p>I have no doubt that the current CTTE and the DPL will fill the
empty seats with worthy project members.</p>
Thoughts on the Init System Coupling GRhttp://upsilon.cc/~zack/blog/posts/2014/11/Thoughts_on_the_Init_System_Coupling_GR/2014-11-20T11:48:31Z2014-11-20T08:59:15Z
<h1>on perceived hysteria and silent sanity</h1>
<p>As you probably already know by now, the results of the Debian
<a href="https://www.debian.org/vote/2014/vote_003">init system
coupling general resolution (GR)</a> look like this:</p>
<div class="center">
<table class="img">
<caption>Init system coupling GR: results (arrow from A to B means
that voters preferred A to B by that margin)</caption>
<tr>
<td><a href=
"http://upsilon.cc/~zack/blog/posts/2014/11/Thoughts_on_the_Init_System_Coupling_GR/results.png">
<img src=
"http://upsilon.cc/~zack/blog/posts/2014/11/Thoughts_on_the_Init_System_Coupling_GR/600x-results.png"
width="600" height="313" alt=
"results of the init system coupling GR" class="img" /></a></td>
</tr>
</table>
</div>
<p>Some random thoughts about them:</p>
<ul>
<li>
<p>The <strong>turnout</strong> has been the <a href=
"https://lists.debian.org/debian-vote/2014/11/msg00192.html">highest</a>
since 2010 DPL elections and the 2nd highest among all GRs (!= DPL
elections) ever. The highest among all GRs dates back to 2004 and
was about <a href=
"https://www.debian.org/vote/2004/vote_002">dropping
<code>non-free</code></a>. In absolute terms this vote scores even
better: it is the GR with the highest number of voters ever.</p>
<p>Clearly there was <em>a lot</em> of interest <em>within</em> the
project about this vote. The results appear to be as representative
of the views of project members as we have been able to get in the
second half of Debian history.</p>
</li>
<li>
<p>There is a total ordering of options (which is not always the
case with <a href=
"https://en.wikipedia.org/wiki/Schulze_method">our voting
system</a>). Starting with the winning option, each option in the
results beats every subsequent option. The winning option ("General
resolution is not required") beats the runner-up ("Support for
other init systems is recommended, i.e., "you <a href=
"https://www.ietf.org/rfc/rfc2119.txt">SHOULD NOT</a> require a
specific init") by a <strong>large margin</strong>: 100 votes,
~20.7% of the voters. The winning options wins over further options
by increasingly large margins: 173 votes (~35.8%) against "Packages
may require specific init systems if maintainers decide" (the MAY
option); 176 (~36.4%) against "Packages may not require a specific
init system" (the MUST NOT option); 263 (~54.5%) against "Further
discussion" (the "let's keep on flaming" option).</p>
<p>While judging from Debian mailing lists and news sites you might
have gotten the impression that the project was evenly split on
init system matters, at least w.r.t. the matter on the ballot that
doesn't seem to be the case.</p>
</li>
<li>
<p>The <strong>winning option</strong> is not as crazy as its label
might imply (voting to declare that the vote was not required?
WTH?). What the winning option actually says is more articulated
than that; quoting from the <a href=
"https://www.debian.org/vote/2014/vote_003#amendmenttextc">ballot</a>
(highlight mine):</p>
<p><em>Regarding the subject of this ballot, the Project affirms
that <strong>the procedures for decision making and conflict
resolution are working adequately</strong> and thus a General
Resolution is not required.</em></p>
<p>With this GR the Debian Project affirms that the procedures we
have used to decide the default init system for Jessie and to
arbitrate the ensuing conflicts are <em>just fine</em>. People
might flame and troll <code>debian-devel</code> as much as they
want (actually, I'm pretty sure we would all like them to stop, but
that matter wasn't on the ballot so you'll have to take my word for
it). People might write blog posts and make headlines corroborating
the impression that Debian is still being torn apart by ongoing
init system battles. But this vote says instead that the large
majority of project members thinks our decision making and
conflict-arbitration procedures, which most prominently include the
<strong>Debian Technical Committee</strong>, have served use
"adequately" well over the past troubled months.</p>
<p>That of course doesn't mean that everyone in Debian is happy
about every single recent decision, otherwise we wouldn't have had
this GR in the first place. But it does mean that we consider our
procedures good enough to (a) avoid getting in their way with a
project-wide vote, and (b) keep on trusting them for the
foreseeable future.</p>
</li>
<li>
<p>[ It is not the main focus of this post, but if you care
specifically about the implications of this GR on
<strong>systemd</strong> adoption in Debian, I recommend reading
<a href="http://www.eyrie.org/~eagle/journal/2014-11/003.html">this
excellent GR commentary</a> by Russ Allbery. ]</p>
</li>
</ul>
<p>My take home message is that we are experiencing a huge gap
between the <strong>public perception</strong> of the state of
Debian (both from within and from without the project) and the
<strong>actual beliefs</strong> of the silent majority of people
that make Debian with their work, day after day.</p>
<p>In part this is old news. The most "senior" members of the
project will remember that the topic of "vocal minorities vs silent
majority" was a recurrent one in Debian 10+ years ago, when flames
were periodically ravaging the project. Since then Debian has grown
a lot though, and we are now part of a much larger and varied
ecosystem. We are now at a scale at which there are plenty of FOSS
"mass-media" covering daily what happens in Debian, inducing
feedback loops with our own perception of ourselves which we do not
fully grok yet. This is a new factor in the perception gap. This
situation is not intrinsically bad, nor there is blame to assign
here: after all influential bloggers, news sites, etc., just do
their job. And their attention also testifies of the huge interest
that there is around Debian and our choices.</p>
<p>But we still need to adapt and learn to take perceived hysteria
with a pinch (or two) of salt. It might just be time for our
decennial check-up. Time to remind ourselves that our ways of doing
things might in fact still be much more sane than sometimes we tend
to believe.</p>
<p>We went on 10+ years ago, after monumental flames. It looks like
we are now ready to move on again, putting The Era of the Great
systemd Histeria™ <a href="https://lwn.net/Articles/619992/">behind
us</a>.</p>
Debsources Participation in FOSS Outreach Programhttp://upsilon.cc/~zack/blog/posts/2014/11/Debsources_Participation_in_FOSS_Outreach_Program/2014-11-17T10:06:32Z2014-11-16T12:45:49Z
<h1>Jingjie Jiang selected as OPW intern for Debsources</h1>
<p>I'm glad to announce that <a href=
"http://sophiejjj.wordpress.com/">Jingjie Jiang</a>, AKA sophiejjj,
has been selected as intern to work on <a href=
"http://sources.debian.net">Debsources</a> as part of the <a href=
"https://opw.gnome.org/">FOSS Outreach Program</a> (formerly known
as Outreach Program for Women, or OPW). I'll co-mentor her work
together with <a href="https://matthieu.io/">Matthieu
Caneill</a>.</p>
<p>I've just added <a href=
"http://sophiejjj.wordpress.com/">sophiejjj's blog</a> to Planet
Debian, so you will soon hear about her work in the Debian
blogosphere.</p>
<p>I've been impressed by the interest that the <a href=
"https://wiki.debian.org/OutreachProgramForWomen#debsources_improvements">
Debsources proposal</a> in this round of OPW has spawned. Together
with Matthieu I have interacted with more than a dozen OPW
applicants. Many of them have contributed useful patches during the
application period, and those patches have been in production at
<a href="http://sources.debian.net">http://sources.debian.net</a>
since quite a while now (see the commit log for details). A special
mention goes to Akshita Jha, who has shown a lot of determination
in tackling both simple and complex issues affecting Debsources. I
hope there will be other chances to work with her in the
future.</p>
<p>OPW internship will begin December 9th, fasten your seat belts
for a boost in Debsources development!</p>
Italy puts Free Software first in public sectorhttp://upsilon.cc/~zack/blog/posts/2014/10/Italy_puts_Free_Software_first_in_public_sector/2014-10-24T19:11:50Z2014-10-24T12:20:59Z
<h1>Debian participation in Italy's CAD68 committee</h1>
<p><small>(The initial policy change discussed in this document is
a couple of years old now, but it took about the same time to be
fully implemented, and AFAIK the role Debian played in it has not
been documented yet.)</small></p>
<p>In October 2012 the Italian government, led at the time by
<a href="https://en.wikipedia.org/wiki/Mario_Monti">Mario
Monti</a>, did something rather innovative, at least for a country
that is not usually ahead of its time in the area of information
technology legislation. They decided to change the <a href=
"http://archivio.digitpa.gov.it/amministrazione-digitale/CAD-testo-vigente">
main law</a> (the "CAD", for <em>Codice dell'Amministrazione
Digitale</em>) that regulates the acquisition of software at all
levels of the public administration (PA), giving an
<strong>explicit preference to the acquisition of Free
Software</strong>.</p>
<p>The new formulation of <a href=
"http://archivio.digitpa.gov.it/amministrazione-digitale/CAD-testo-vigente#art68">
article 68</a> of the CAD first lists some macro criteria (e.g.,
<a href=
"https://en.wikipedia.org/wiki/Total_cost_of_ownership">TCO</a>,
adherence to open standards, security support, etc.) that public
administrations in Italy shall use as ranking criteria in
software-related calls for tenders. Then, and this is the most
important part, the article affirms that the acquisition of
<em>proprietary</em> software solutions is permitted <em>only</em>
if it is <em>impossible</em> to choose Free Software solutions
instead; or, alternatively, to choose software solutions that have
already being acquired (and paid for) by the PA in the past,
reusing preexisting software. The combined effect of these two
provisions is that all <em>new</em> software acquisitions by PAs in
Italy will be Free Software, unless it is motivated—in writing,
challengable by a judge—that it was impossible to do otherwise.
Isn't it great?</p>
<p>It is, except that such a law is not necessarily easy to adhere
to in practice, especially for small public administrations (e.g.,
municipalities of a few hundred people, not uncommon in Italy)
which might have very little clue about software in general, and
even less so about Free Software. This is why the government also
tasked the <a href="http://www.agid.gov.it/">relevant Italian
agency</a> to provide guidelines on how to choose software in a way
that conforms with the new formulation of article 68. The agency
decided to form a committee to work on the guidelines (because you
<em>always</em> need a committee, right? <img src=
"http://upsilon.cc/~zack/smileys/smile.png" alt=":-)" /> ).</p>
<p>To my surprise, the call for participation to be part of the
committee explicitly listed representatives of Free Software
communities as privileged software stakeholders that they wanted to
have on the committee—kudos to the agency for that. <small>(The
Italian wording on the call was: <q>Costituirà titolo di preferenza
rivestire un ruolo di […] referenti di community del software a
codice sorgente aperto.</q>)</small> Therefore, after various prods
by fellow European Free Software activists that were aware of the
ongoing change in legislation, I applied to be a volunteer CAD68
committee member, got selected, and ended up working over a period
of about 6 months (March-September 2013) to help the agency writing
the new software acquisition guidelines.</p>
<p>Logistically, it hasn't been entirely trivial, as the default
meeting place was in Rome, I live in Paris, and the agency didn't
really have a travel budget for committee members. That's why I've
sought sponsorship from Debian, offering to represent Debian views
within the committee; Lucas kindly <a href=
"https://lists.debian.org/debian-devel-announce/2013/05/msg00002.html">
agreed</a> to my request. So what did I do on behalf of Debian as a
committee member during those months?</p>
<p>Most of my job has been some sort of consulting on how
community-driven Free Software projects—like Debian—work, on how
the software they produce can be relied upon and contributed to,
and more generally on how the PA can productively interact with
such projects. In particular, I've been happy to work on the
related work section of the guidelines, ensuring they point to
relevant documents such as the French government guidelines on how
to adopt Free Software (AKA <a href=
"http://circulaire.legifrance.gouv.fr/pdf/2012/09/cir_35837.pdf"><strong>
circulaire Ayrault</strong></a>). I've also drafted the guidelines
section on Free Software directories, ensuring that important
resources such as <a href=
"https://directory.fsf.org/"><strong>FSF's Free Software
Directory</strong></a> are listed as starting points for PAs that
looking for software solutions for specific needs.</p>
<p>Another part of my job has been ensuring that the guidelines do
not end up betraying the principle of Free Software preference that
is embodied in article 68. A majority of committee members came
from a Free Software background, so that might not seem a difficult
goal to accomplish. But it is important to notice that: (a) the
final editor of the guidelines is the agency itself, not the
committee, so having a "pro-Free Software" majority within the
committee doesn't mean much per se; and (b) lobbying from the
"pro-proprietary software" camp did happen, as it is entirely
natural in these cases. In this respect I'm happy with the result:
I do believe that the software selection process recommended by the
guidelines, finally <a href=
"http://www.agid.gov.it/notizie/riuso-valutazione-comparativa-online-la-circolare">
published</a> in January 2014, <strong>upholds the Free Software
preference principle</strong> of article 68. I credit both the
agency and the non-ambiguity of the law (on this specific point)
for that result.</p>
<p>All in all, this has been a positive experience for me. It has
reaffirmed my belief that Debian is a respected, non-partisan
political actor of the wider software/ICT ecosystem. This
experience has also given me a chance to be part of country-level
policy-making, which has been very instructive on how and why good
ideas might take a while to come into effect and influence citizen
lives. Speaking of which, I'm now looking forward to the first
alleged <em>violations</em> of article 68 in Italy, and how they
will be dealt with.</p>
<p>Abundant popcorn will certainly be needed.</p>
<h2>Links & press</h2>
<p>If you want to know more about this topic, I've collected below
links to resources that have documented, in various languages, the
publication of the CAD68 guidelines.</p>
<ul>
<li>official material:
<ul>
<li><a href=
"http://www.agid.gov.it/notizie/riuso-valutazione-comparativa-online-la-circolare">
announcement</a> (it) by the agency</li>
<li><a href=
"http://www.agid.gov.it/sites/default/files/linee_guida/circolare_agid_63-2013_linee_guida_art_68_del_cad_ver_13_b.pdf">
guidelines</a> (.pdf, it)</li>
</ul>
</li>
<li>press releases:
<ul>
<li><a href=
"https://fsfe.org/news/2014/news-20140116-01.en.html">FSFE press
release</a> (en), also <a href=
"https://lwn.net/Articles/580591/">covered by LWN</a> (en)</li>
<li><a href=
"http://www.april.org/litalie-met-en-place-la-priorite-pour-le-logiciel-libre-dans-ladministration">
April press release</a> (fr), also <a href=
"http://www.april.org/en/italy-implements-prioritization-free-software-public-administration">
in english</a></li>
</ul>
</li>
<li>press/blog coverage:
<ul>
<li><a href=
"https://joinup.ec.europa.eu/community/osor/news/italy-posts-benchmark-open-vs-closed-software">
on Joinup</a> (en), European Commission blog dedicated to Free
Software adoption in Europe, with comments by yours truly and Carlo
Piana</li>
<li><a href=
"http://opensource.com/government/13/11/free-open-source-italian-public-administration">
on OpenSource.com</a> (en), popular blog for FOSS-related
topics</li>
<li><a href="http://piana.eu/it/68cad">by Carlo Piana</a> (it),
fellow member of the CAD68 committee, representing KDE and
FSFE</li>
<li><a href=
"http://punto-informatico.it/3973355/PI/News/pa-perche-scegliere-software-foss.aspx">
on Punto Informatico</a> (it), popular Italian tech blog</li>
<li><a href=
"http://www.apogeonline.com/webzine/2014/01/15/se-non-basta-la-legge-ecco-pronte-le-linee-guida">
on Apogeo online</a> (it), blog of a popular Italian publisher of
IT books</li>
<li><a href=
"http://www.agendadigitale.eu/egov/623_le-norme-per-il-software-libero-nella-pa-pro-e-contro.htm">
on Agenda Digitale EU</a> (it), blog dedicated to digital
innovation in the Italian PA</li>
<li><a href=
"http://openisfree.blogspot.fr/2014/01/italian-foss-guidelines.html">
by Simone Aliprandi</a> (en), lawyer who followed closely the work
of CAD68 committee</li>
</ul>
</li>
<li>interviews:
<ul>
<li><a href=
"http://www.mysolutionpost.it/blogs/it-law/piana/2014/01/acquisti-software-pa.aspx">
on My Solution</a> (it), by yours truly</li>
<li><a href=
"http://caterpillar.blog.rai.it/2014/01/20/caterpillar-del-20-gennaio/">
radio interview</a> (it) by yours truly for <a href=
"http://caterpillar.blog.rai.it/">Caterpillar</a>, RAI Radio 2</li>
</ul>
</li>
</ul>
debsources bugs and easy hackshttp://upsilon.cc/~zack/blog/posts/2014/09/debsources_bugs_and_easy_hacks/2014-09-11T17:31:13Z2014-09-11T17:31:13Z
<h1>debsources debbugs oh</h1>
<p>My <a href=
"http://upsilon.cc/~zack/blog/posts/2014/08/debsources_hacking/">ongoing quest</a>
for lowering the barrier for contributing to <a href=
"http://sources.debian.net">Debsources</a> continues.<br />
In this chapter:</p>
<ul>
<li>
<p>I've migrated <strong>bug reports</strong> from the previous
ad-hoc text file in the Git repo to the <a href=
"https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=qa.debian.org;tag=debsources">
<strong>Debian BTS</strong></a>, under the umbrella of the
<code>qa.debian.org</code> pseudo-package.<br />
From now on this is the recommended (and <a href=
"http://sources.debian.net/about/">documented</a>) way of reporting
bugs against <a href=
"http://sources.debian.net">http://sources.debian.net</a>.</p>
<p>Look ma, it also has one of those newfangled short URLs:
<a href="http://deb.li/debsrcbugs">http://deb.li/debsrcbugs</a>!</p>
</li>
<li>
<p>While at it, I've also properly tagged the current <a href=
"https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=qa.debian.org;include=subject:debsources;users=debian-qa@lists.debian.org;tag=gift">
<strong>easy hacks</strong></a> on Debsources using the <a href=
"https://wiki.debian.org/qa.debian.org/GiftTag">gift tag</a>. There
are definitely opportunities for new contributors there, and there
might be more if you submit your own Debsources' pet peeves to the
BTS.</p>
<p>Again, mandatory mnemonic/short URL: <a href=
"http://deb.li/debsrceasy">http://deb.li/debsrceasy</a>.</p>
</li>
</ul>
<p>What's your excuse for not contributing to Debsources,
again?</p>
debsources paper at ESEM2014http://upsilon.cc/~zack/blog/posts/2014/06/debsources_paper_at_ESEM2014/2014-06-06T11:22:34Z2014-06-06T11:22:34Z
<h1>Debsources: Live and Historical Views on Macro-Level Software
Evolution</h1>
<p>The paper entitled <a href=
"http://upsilon.cc/~zack/research/publications/debsources-esem-2014.pdf"><em>Debsources:
Live and Historical Views on Macro-Level Software
Evolution</em></a>, which I've co-authored with Matthieu Caneill,
has been accepted at <a href=
"http://softeng.polito.it/ESEIW2014/ESEM/">ESEM 2014</a>: the 8th
international symposium on Emprical Software Engineering and
Measurement.</p>
<p>In the paper we have described Debsources as a software platform
for monitoring the evolution of Free Software through the lenses of
Debian, and used the main Debsources instance (<a href=
"http://sources.debian.net">http://sources.debian.net</a>) to
replicate and extend a <a href=
"http://link.springer.com/article/10.1007/s10664-008-9100-x">former
study</a> on macro-level software evolution.</p>
<p>Now we "just" have to integrate all the nice charts and data we
have extracted for the paper into Debsources' <a href=
"http://sources.debian.net/stats/">stats page</a>...
<code>/o\</code></p>
historical overview of debian source codehttp://upsilon.cc/~zack/blog/posts/2014/04/historical_overview_of_debian_source_code/2014-06-06T11:29:49Z2014-04-06T11:19:12Z
<h1>moar, and moar, and moar debsources stats</h1>
<p>A while ago I've <a href=
"http://upsilon.cc/~zack/blog/posts/2014/02/moar_stats_for_sources.debian.net/">announced</a>
the availability of <a href=
"http://sources.debian.net/stats/">several stats</a> about Debian
source code on <a href=
"http://sources.debian.net">http://sources.debian.net</a>. Since
then the statistical basis of those stats has increased a lot, and
now includes <strong>all Debian historical releases</strong>, from
<a href="https://www.debian.org/releases/hamm/">hamm</a> (July
1998) onward. This allows to appreciate macro-level evolution
trends in Free Software, over a period of more than 15 years,
through the eyes of a distro that sits at the nice intersection of
the eldest, largest, and most reputed distros.</p>
<p>To get there I've added support for <strong>sticky
suites</strong> to the plumbing layer of <a href=
"http://anonscm.debian.org/gitweb/?p=qa/debsources.git">debsources</a>,
and then injected historical releases from <a href=
"http://archive.debian.org">http://archive.debian.org</a>. The
injection process took about a week (without any sort of
parallelism, pretty slow disks, and computing sha256 checksums,
ctags, and sloccount on all source files) and has been an
"interesting" experience.</p>
<p>When you go back decades in technology time, <strong>bit
rot</strong> is just around the corner, and I've found <a href=
"https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740883">my</a>
<a href=
"https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741012">share</a>
while injecting <code>archive.d.o</code> into
<code>sources.d.n</code>. In both cases the respective maintainers
(Guillem and Ganneff, kudos) have been positive about and helpful
in improving the situation, despite the low impact of the bugs I've
found on the average user. That's quite important for the
<strong>long-term preservation</strong> of digital information in
general, and for the perennity of access to Free Software in the
specific case of Debian.</p>
<p>While we are it, I'm now maintaining a list of <a href=
"https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=debsources;users=zack@debian.org">
bugs affecting <code>sources.d.n</code></a> but belonging to other
packages, in case you fancy helping out but are not a Python
hacker. Interestingly enough, quite a bit of those bugs are related
to the fact that tools debsources uses (e.g. ctags, sloccount) are
also starting to show their age.</p>
<p>You might wander why <a href=
"https://www.debian.org/releases/buzz/">buzz</a>, <a href=
"https://www.debian.org/releases/rex/">rex</a>, and <a href=
"https://www.debian.org/releases/bo/">bo</a> are still missing from
<code>sources.d.n</code>. That's in fact for similar reasons.
Before hamm Debian didn't have complete archive coverage in terms
of <code>Sources</code> indexes and <code>.dsc</code> files. Given
that debsources rely on both to extract source packages, it first
needs to grow an additional abstraction layer that can cope with
their absence. It's SMOP, and planned.</p>
<p>And now let's have fun with <a href=
"https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=742605">ctags
bombs</a>.</p>
<p>Yours truly,<br />
Stefano “Indiana” Zacchiroli<br />
<small>(credits: KiBi, <code>#debian-ftp</code>)</small></p>
moar stats for sources.debian.nethttp://upsilon.cc/~zack/blog/posts/2014/02/moar_stats_for_sources.debian.net/2014-02-27T19:31:05Z2014-02-27T19:22:00Z
<h1>Debian: watch your stats!</h1>
<p>Over the past few weeks, myself and Matthieu Caneill have worked
quite a bit on <a href=
"http://anonscm.debian.org/gitweb/?p=qa/debsources.git"><strong>Debsources</strong></a>.
As we have now deployed most of the new features on <a href=
"http://sources.debian.net">http://sources.debian.net</a>, it's
time for another <em>"What's new with Debsources?"</em> blog post.
Here is what's new:</p>
<ul>
<li>
<p>Debsources now knows about Debian <strong>suites</strong>, i.e.
which package is in which "release" (stable, testing, unstable,
...). This knowledge is already useful for some of the other
features below and will be used more in the future.</p>
</li>
<li>
<p><a href=
"http://upsilon.cc/~zack/blog/posts/2013/09/sources.debian.net_-_advanced_search_and_other_news/">
since last summer</a> Debsources has been running
<strong>sloccount</strong> on all unpacked source packages,
together with ctags and du, but the resulting information wasn't
exposed on the Web. This is now fixed. Each package now has an
<strong>infobox</strong> (<a href=
"http://sources.debian.net/src/linux/3.2.54-2">example</a>) which
shows: disk usage, archive area, suites, and sloccount with
per-language breakdown. The new infobox also subsumes the old puny
list of package links.</p>
<p>You can easily embed the infobox in other webapps if you need to
(<a href=
"http://sources.debian.net/embed/pkginfo/linux/3.2.54-2/">example</a>).
Check the <a href="http://sources.debian.net/doc/url/">URL scheme
doc</a> for more info.</p>
</li>
<li>
<p>Debsources now gathers and plot accurate Debian sources <a href=
"http://sources.debian.net/stats/"><strong>statistics</strong></a>,
both overall and per-suite, in both snapshot and <strong>historical
trends</strong> flavors.</p>
<p>(Yeah, I know, the charts are not particularly good looking ATM,
but that's easy to change without impacting the rest. So if you're
a <a href="http://matplotlib.org/">matplotlib</a> artist and
willing to help, please step forward!)</p>
</li>
<li>
<p>many changes have been going on also at the
<strong>plumbing</strong> layer to make the service less resource
hungry and more maintainable, in view of a migration to the
official Debian infrastructure --- which I've in the meantime
started discussing with DSA. Some highlights:</p>
<ul>
<li>
<p>Debsources now has a rather comprehensive <strong>test
suite</strong>, built using <a href=
"https://nose.readthedocs.org/en/latest/">Nose</a>. Most notably,
we do test full update runs down to source unpacking (of a small
subset of a Debian mirror), DB injection, and plugin execution ---
which is quite neat.</p>
</li>
<li>
<p>the updater is now much faster (about 2x) and might require, in
pathological cases, 10x <em>less</em> memory than before. Memory
usage now caps at around 300MB, even when injecting ctags for large
packages such as linux, chromium, and libreoffice.</p>
</li>
<li>
<p>the DB schema went through several refactoring cycles, and now
uses a separate <strong>file table</strong> to index all known
source file paths. In the past path information were duplicated
across the checksums and ctags tables, not only wasting DB space,
but also making the presence of file information conditional on the
enablement of at least one of the two corresponding plugins. This
is now fixed --- and migrating the full DB has been quite "fun".
Unfortunately, we've also added quite a few large-ish indexes,
resulting in no significant overall changes in DB size (currently
at ~50GB), but at least in much faster queries <img src=
"http://upsilon.cc/~zack/smileys/smile.png" alt=":-)" /></p>
<p>The next step on this front will be the addition of path-based
searches, using the excellent Postgres <a href=
"http://www.postgresql.org/docs/9.1/static/pgtrgm.html">trigram
indexes</a>.</p>
</li>
</ul>
</li>
</ul>
<p>Want more? Sure, we'll be happy to! But it'll happen faster if
you <a href=
"http://anonscm.debian.org/gitweb/?p=qa/debsources.git;a=blob;f=BUGS;hb=refs/heads/bugs">
help</a>. Speaking of which: we've got Debsources into the <a href=
"https://wiki.debian.org/DebianFrance/NewContributorGame"><strong>new
contributors game</strong></a> (see <a href=
"https://lists.debian.org/debian-devel-announce/2014/02/msg00009.html">
announcement</a>) and we're looking forward to mentor new
contributors.</p>