Alan Harnum

Unsolicited Technical Advice for Toronto Public Library From Someone Who Used to Work There

14 Sep 2016

Introductory Remarks

Bianca Wylie published a recent article on Torontoist about the problems with procurement practices in government technology.

I had this to say when I retweeted the article:

I saw all of the issues here up close during the 6 years I worked in tech for Toronto Public Library. https://t.co/RZKNqMJz6X

— Alan Harnum (@waharnum) September 3, 2016

It goes without saying that my perceptions of the strengths and deficiencies of how libraries deploy and manage technology was shaped most specifically by the ten years I worked at Toronto Public Library, six of them with the library's web services group. Much of my general interest in how public sector organizations use and misuses technology arises out of my attempts to frame the understanding of my workplace experience more broadly.

So that Twitter comment deserves expansion, in a more public, permanent and detailed space than Twitter. The more involvement I have with groups like Civic Tech Toronto, the more I feel that one responsibility of technologists who have left public service partly because of their frustrations with technology issues in a particular public service is to speak out publicly about it, to give a more concrete weight to the issues of a particular place and time. Government and its agencies are held to account (eventually, theoretically, sometimes) by the public, but the public's perspective is limited by information asymmetry. So there is a moral imperative from those of who were inside but are outside now to speak out; we are both of that public, and not yet entirely outside.

The general issues I raise are not unique to TPL among libraries, a sector that has had to respond and adapt to the digital age without any reduction in previous service delivery requirements and in times of ever-increasing funding uncertainty. They also apply generally to most government and government agencies. Above above all else I say what I say with love and appreciation for the many things TPL does right.

I'll preface what I have to say with the following additional caveats:

Mediocre/Outdated IT & Problematic Procurement Requirements

In the Torontoist article, Wylie says the following (emphasis mine):

Sadly, what’s most damaging to the quality of core service delivery from our governments are the countless mediocre and outdated IT systems and solutions that we’ve been buying, building, and maintaining for decades. They don’t blow up or make headlines. They cost hundreds of millions annually. They are the direct result of outdated procurement processes. And they fail to improve lives daily. We should be angrier about this.

And also this:

...tight legislation around purchasing, compounded by problematic requirements from internal IT, lead governments to issue requests for solutions that are as specific as they have to be, but vague where they can be.

The whole article is very much worth reading, and there's much more than the below to say about technology at TPL, but I'll use those two quotes as my jumping off points here.

"Countless Mediocre and Outdated IT Systems and Solutions That We've Been Buying, Building and Maintaining for Decades"

This was a significant issue in the time I worked for the library. Off the top of my head (any dates are approximate), two examples:

The Endeca-based Website

The underlying search technology powering the website is a custom solution based on Endeca. Endeca was an innovative proprietary search technology when the development of the platform began in 2008 (I joined the web group in early 2009, when development was already substantially underway). Since then, Endeca was acquired by Oracle (in 2011), and gone through a corresponding realignment towards e-commerce and away from search.

I can say from firsthand experience that development with Endeca is as laborious, expensive and requiring of pricey consultants as would be expected from an Oracle product. The library made significant strides forward (particularly following the problematic switch from the Dynix integrated library system to the Symphony one, a series of events that deserve their own blog post) with its digital presence when the site was launched, but it's showing its age, new features are difficult to develop, and various open-source search technologies have come of age in the last five years, both in and outside the library space. Specifically, I am thinking of projects like ElasticSearch in the general search space, and Blacklight in the library discovery space.

Search and discovery are areas where TPL should legitimately be investing heavily in custom development and ongoing maintenance, but there are better ways to do it these days than Endeca.

Custom-Built Digitization System and Workflow

I know less about these systems specifically, but I do know that the systems used in the digitization workflow of the library's special collections are, like the website's search, based on proprietary Oracle technology (an enterprise CMS system); I am talking here not about the public search and display of items, but the systems by which collection is scanned, catalogued and archived before delivery to the Endeca-based systems.

I believe the original implementation dates from the late 90s or early 2000s. This may have been a reasonable decision for the time, given the overall state of library digitization systems in those days; I wasn't there and can't comment.

My sense from the periphery when I worked for the library is that this system was no longer serving the library well in terms of efficiency, features, maintainability or possibilities for enhancement. This also comes at a time when the open source solutions for digital archiving like Islandora and Hydra are at a significant level of maturity.

Related Suggestions to Consider

  1. TPL should be more judicious about highly custom solutions, as opposed to the adaptation of software and SaaS more targeted at the specific problem it is trying to solve or the service it is trying to deliver. I'd characterize this as "bounded custom development", and to that end...
  2. Custom development, when done, should be done on open source technology, and ideally be open source itself (the TPL E-Services GitHub repository should not be so empty). This is not only based on aligning technology to the library's stated values, but on a more pragmatic perspective of lowering custom development cost and complexity.
  3. When the time comes to replace an existing system, a baseline goal should be to lower the cost and complexity of operating and maintaining the system over its expected lifetime, both in terms of staff work on the system, and the involvement of external consultants. There was an extremity of bailing water-style systems management when I worked for TPL, vs plugging leaks, and many smaller-scale systems hosted and supported internally that could very likely be replaced with SaaS platforms at lower cost in time and money.

"Compounded By Problematic Requirements from Internal IT, Lead Governments to Issue Requests for Solutions That Are as Specific as They Have to Be, But Vague Where They Can Be"

When I worked for TPL, it was a significant struggle (both politically and technically) to use any web technology not based on Java code running on Solaris SPARC machines. Combined with conservative (even for government) attitudes about both the cloud and open-source software, this significantly constrained the possible range of technology solutions. This impacted both the RF(P/Q/I) process and purely internal systems management and development.

I expended significant technical effort and political capital in my last six months at TPL executing some projects in the cloud, with what I consider reasonable success, and before that on making sure the Account replacement project was developed using Ruby, with what I consider much less success (another topic that really deserves its own post, including how I became possibly the world expert on trying and failing to compile a modern version of CRuby-based Rails on Solaris SPARC architecture).

My general feeling during the time I worked for TPL was that too much money and effort was being spent on implementing comparatively expensive infrastructure such as Solaris SPARC systems for an era when datacentre operations have generally shifted to commodity x86 servers running various flavours of Linux, or into the cloud, just as too much money was being spent on custom development for what was being delivered to the library's patrons.

Related Suggestions to Consider

  1. TPL needs to get into the cloud in a much more serious way. At this point, compared to the New York Public Library, they will be in the comparatively low-risk position of a late adopter. Many cloud vendors have opened onsite Canadian data centres, making it easier to address concerns about patron data and privacy.
  2. Commodity x86 hardware and some variant of Linux should be the rule for any on-site datacentre servers. Probably Red Hat, given the use of Red Hat software and support in other contexts by TPL (the main website runs on JBoss).
  3. TPL should have its own outside review conducted of its technology platform, if not intended or invited to be a participant in the probably-upcoming Assessment and review of the City of Toronto's information technology platform.
  4. TPL should disclose more public details about what it holistically spends on technology and where the money is spent, including specific details of how the 1.4 million Virtual Reference Library operating grant received from the Government of Ontario is allocated.

Conclusion

I'll conclude with the concluding quote from Wylie's article:

The overhaul needed in the Canadian context requires updates to laws that govern purchasing. In an ideal world, the laws that dictate government data management and sharing practices would be be revisited in parallel. By failing to delve into these pre-tech-era laws that force governments and their vendors into buying and building bad solutions, we’re destined to make the same mistakes time and time again.

I agree entirely on this point, but I will also add that a pervasive problem for government agencies is the maladaptive culture around technology that develops when people manage and work with technology for a long time under these conditions. An ingrained self of helplessness, futility and dependence on outside vendors (both to implement the bad solutions, and then be blamed for the bad solutions not meeting public or staff expectations) develops, with serious consequences for the entire public services organization and those it delivers services to.

Addressing that culture issue begins with making different technical choices to the extent possible within existing purchasing rules that, over time, will increase an agency's ability to deliver more and more effective digital services. While hands are somewhat tied, they are certainly not tightly bound; there are no overnight solutions (or silver bullets), but there are directions that are better than others, better than the status quo, and fully achievable under current procurement rules.

At this point we know some of the general shape for libraries and government agencies to start taking more control of their technical destiny and taking fuller responsibility (no more blaming vendors) for delivering excellent digital services. We have examples like:

That future looks like more RF(I/P/Q)s specifying open source and cloud, more investment in internal staff working with those technologies, more open data and APIs to make it easier to hack on government services; above all, it looks like a refusal to continue participating in 2016 in a dysfunctional relationship with technology vendors shaped by 20th century procurement practices.