Looking at National Portfolio Orgs, and some questions from a friend

In my continuing quest for a bird’s eye view on the arts & culture landscape, today I poked at the current data on the Arts Council’s National Portfolio Organisations (NPO). On the Arts Council website, there’s an Excel download of all the NPO organisations for 2018-2022. So, I downloaded that, and made a Google spreadsheet version: Copy of all NPOs and funding 2018-2022.

Copy of all NPOs and funding 2018-2022

I found this useful to help with grounding amongst all the big numbers flying around. It helped me realise that the £1.57M relief promised to the sector is a big amount, since that’s just below the total funding across all NPOs for 2018-2022, which amounts to £1,622,621,636. The difficult part is that there are 840 organisations listed here, and — even though I’m not sure anyone actually knows — there are lots more than 840 arts orgs across the UK.

The disciplines listed are:

  • Combined arts
  • Dance
  • Literature
  • Museums
  • Music
  • Theatre
  • Visual arts, and
  • Not specific

When I sorted the sheet by Column 0 (% change in support from one period to the other) + Column D (Funding band) + Column N (Total Funding 2018-2022), you quickly see:

  • 187 orgs have been added for four-year period, adding £147,487,464 to the funding pot
  • Without Walls and Arts & Heritage Plymouth City Council are the big new NPOs in terms of £, each with over £1,000,000 investment each year – yay!
  • 526 orgs have retained the same level of funding across both periods, 0% change
  • You can see little blobs of new NPOs at round funding figures; like seeds

Some other facts I didn’t know before poking at this data:

  • Royal Opera House is funded to the tune of £24,028,840 annually (and that’s about 20% of its income normally)
  • There are 36 “Band 3” orgs, which receive a minimum of £1,000,000 per year, and have other requirements around diversity, international collaboration, education, digital etc, per Page 41 of the NPO Relationship Framework.
  • There are 58 Sector Support Organisations (SSOs), including 24 new ones in this period. SSOs are great.

I found this table not very helpful in the ACE-produced NPO info, perhaps because it doesn’t connect with the overall list of NPOs or show crossover, for example a BME+LGBT+Female-led? Or, am I missing something blindingly obvious here?

ACE NPO Investment Factsheet PDF

What are the big juicy questions?

The questions from my friend, and long time advisor and conversator from afar, Eliza Gregory, has been chatting and emailing with me about all this… and she’s sent me questions that I like so much, I’m just going to copy them in here.

  1. What are the social implications of entrenching cultural wealth in terms of endowments, grants and artifacts?
  2. What social problems are connected to these practices?
  3. And on the other side, what might society be like if we changed the way a portion of that wealth is held or distributed?
  4. In what ways do these practices uphold the colonial project/the colonial/Victorian legacy that actually holds the entire society back?
  5. In what ways do these practices continue to explicitly support white supremacy?
  6. And what would it mean for all people to be liberated from that structure?
  7. Can we picture ourselves freed from it?
  8. What does that look like?
  9. What does that feel like? (So, so good.) 

Again, rough stuff, comments welcome, suggestions for direction or collaborators who are doing similar things even more welcome!

Arm’s Length Data?

I’ve spent today looking up the 2019 Annual Reports of all the Arm’s Length Organisations that DCMS gives grant-in-aid funding to. I’ve put some figures I think are involved in my learning about Who Needs It Less?

I’ve made a Google spreadsheet called DCMS Arm’s Length Orgs snapshot – 2018-2019, and anyone can view with this link. Please tread carefully! THERE MAY BE ERRORS. This is rough work, comments welcome!

These are big numbers. I’ve collated a few different fields per organisation I think are interesting:

  • Total income (for that year)
  • Total expenditure
  • Net income (which may be in the red)
  • Fixed assets (which likely include tangible, intangible, heritage, and certain types of investments)
  • Current assets (like stocks, debtors or cash)
  • Grant-in-aid figure if I could find it in the Annual Report (and there’s a second sheet in the spreadsheet that gets that number per org from a doc I found from Parliament, which is linked as the source)
  • Endowment if that figure is noted separately

From this basic by-hand aggregation, you can see stuff like the BBC’s Total Income in 2018 was £4,889,000,000 or National Gallery had the highest Net Income at £15,400,000.

Then I added two % calculations:

  1. Current Assets as a percentage of the Total Income for that year, and
  2. Grant-in-aid as a percentage of the Total Income for that year

Now, I’m not stating anything resembling an approach to trying to figure out which orgs to support and how, but, I’m wondering about these two percentage figures… could they be some measure of health or stability? As Frankie rightly commented on my previous post about this, the Fixed Assets held by our great institutions are probably basically irrelevant, since they’re practically priceless. But maybe if you can say something like the Imperial War Museum has Current Assets that could cover about 64% of its annual income, does that get us anywhere? Or that Royal Armouries has 12% coverage from its Current Assets?

What if we look for orgs that have current (or, more fluid) assets that cover less than 20% of their annual income for 2018 and help them first? Or 50%? Better yet, we could filter that list to deliberately favour BAME and LGBT and disabled-led orgs.

What if the government (and our society) is able to seize this moment to actively work against the preferential structures in its own system? It could actively generate assets for littlies. Grant them 1-3 years equivalent to their 2018 income, and give them an endowment equal to the average of the Arm’s Length orgs, which by my rough calculations is 47% of 2018 income in the bank. That would be a reflection of the healthy situation DCMS has built with their Arm’s Length program, would it not?

I thought I’d have a look at NPOs next, poking at that Current Assets idea. It can be enlightening to see who has no wealth, when that’s such a marker of systemic exclusion.

Notes on data creation:

  • I’ve left comments on cells if something odd or there’s extra info or detail
  • Sources are individual org’s annual reports, linked in Column B
  • If there’s an overarching group, I’ve used that number
  • Director’s Pay is the total package, salary + pension etc
  • DCMS grant in aid is as noted in the annual report
  • I’ve basically looked for what appears to be the same numbers across all the annual report documents – that’s mostly the Balance Sheet and Financial Statements
  • If I’ve left a cell (or row, in the case of the BBC) blank, that means it’s too hard for me to find or process or put into this structure

Looking at UK Arts & Culture funding

Introduction

This post is a branching off from my (George) personal blog, where I have been writing throughout lockdown about various things. I am interested to try to teach myself more about the Arts & Culture funding landscape in the UK, especially as the government has just announced their £1.57M grant, and because I’ve been trying harder to see where systemic racism and sexism live. I want to know how the funding will be distributed. I’m curious to see if I can draw the overall arts and culture funding picture a bit more clearly for myself, and thought others might be interested.

I would love for this to a be a conversation, especially if I’m really missing very important aspects as I explore. Comments are welcome.

Assets vs Need?

Today I haven’t been able to stop thinking of something that struck me when I first moved to the UK in 2014. One day, fairly soon after I’d moved here, I happened to have a coffee with Ed Vaizey, with my friend Wolfgang. He was very pleasant. I was not at all prepared as well as I should have been, and nothing came of it, which is one of my few regrets. But, the thing I noticed and remember most was that Mr. Vaizey’s shirt collar was frayed. How strange, I thought, that the Minster for Digital and Creative Industries couldn’t afford a shirt that wasn’t frayed. I mean, he has his own coat of arms.

Ed Vaizey’s coat of arms

I remember mentioning this to a new English friend who informed me that this was an ever so subtle class marker. That upper class people like to wear things out instead of buying new replacements. Very, very wealthy people apparently don’t have much actual cash, since all their wealth is tied up in things that are difficult to extract their wealth from, like a big house, or, say, most of the real estate in Bloomsbury, as is the case with the British Museum, whose total net assets were listed as £1,001,693,000 in its 2018/19 Annual Report on the Consolidated Balance Sheet, as at 31 March 2019. How hard it must be to see all that money listed as a line item in a balance sheet and not be able to use it.

I’ve been thinking about that £1,570,000,000 cash injection offered by the government to the arts sector, and trying to think about Who Needs This Funding The Least? It’s early days for my data gathering and poking, and sadly, the decisions have likely already been made about who is going to benefit, although I understand there will be some form of application for some. I’ve found it easy to let the various giant numbers flying around wash over me… 1 billion here, 120 million there so step one is to try to see some of these numbers, and particularly to see them against other comparators, to get a sense of the scale of the situation.

Today I learned that the Department of Culture, Media and Sport (DCMS) gives funds each year to what’s called “Arm’s Length Bodies” which receive what’s called “Grant in Aid”. I found a DCMS Estimate Memorandum containing a certain Table 3: Comparing the Grant in Aid funding of our Arm’s length bodies in 2016-17 through to 2018-19, which I share with you below: 

Organisation2017 projected2017 actual2018 projected
British Broadcasting Corporation£3,156,700,000£3,185,400,000£3,255,500,000
Arts Council England£460,526,000£494,183,000£479,972,000
Ofcom1£72,295,000£123,039,000£106,300,000
Sport England£105,649,000£101,787,000£104,795,000
British Library£93,911,000£93,893,000£93,443,000
Historic England£87,806,000£87,912,000£90,734,000
UK Sport£53,536,000£60,890,000£61,431,000
VisitBritain£56,972,000£60,458,000£56,818,000
British Museum£53,569,000£53,473,000£42,046,000
Natural History Museum£49,115,000£41,815,000£41,815,000
Science Museum Group£43,343,000£46,903,000£40,428,000
Tate Gallery£40,251,000£38,066,000£37,566,000
Victoria & Albert Museum£40,257,000£37,726,000£37,176,000
National Gallery£24,092,000£24,092,000£24,092,000
Imperial War Museum£32,136,000£25,347,000£23,634,000
British Film Institute£23,965,000£23,587,000£20,878,000
National Museums Liverpool£20,050,000£19,761,000£19,761,000
Royal Museums Greenwich£16,019,000£16,019,000£15,869,000
S4C3£6,762,000£6,956,000£15,097,000
Royal Armouries£7,088,000£7,788,000£8,938,000
UKAD£6,096,000£6,046,000£7,998,000
National Portrait Gallery£6,637,000£9,734,000£6,634,000
National Heritage Memorial Fund£35,250,000£5,489,000£5,000,000
Horniman Museum and Gardens£4,549,000£4,320,000£3,820,000
Information Commissioners Office£3,790,000£5,740,000£3,750,000
The Wallace Collection£2,711,000£3,711,000£2,711,000
Churches Conservation Trust£2,749,000£2,738,000£2,604,000
Geffrye Museum£1,696,000£1,786,000£1,796,000
Sports Grounds Safety Authority 2£0£1,542,000£1,601,000
Sir John Soane’s£1,983,000£1,012,000£1,012,000
Table 3: Comparing the Grant in Aid funding of our Arm’s length bodies in 2016-17 through to 2018-19

Isn’t that interesting? That is a bunch of support. What robust affirmative action! A total of £4,613,219,000 projected to be granted to these 30 “arm’s length” organisations in 2018. There’s the British Museum up there in the list, which was projected to receive £42,046,000 in 2018. The BM’s annual report (linked above) confirms for us that indeed: “The British Museum received £39.4 million revenue and £13.1 million capital grant-in-aid from the DCMS in 2018/19” on page 16.

All 30 organisations who receive this Grant in Aid are required to sign a Management Agreement with DCMS, and report back in a standard way so DCMS can see how well the grants are being used and measure performance consistently. For example, here is the Total income of DCMS-funded cultural organisations 2018/19 report from DCMS.

Big numbers can be numbing

The government’s support package announced this week to be spread across lots more organisations is about 34.03% of that total annual “arms length” grant in aid dispensed in 2018. I hope my maths is correct, otherwise I’m going to look even more naive and foolish. I am very willing to be called out on this if I have made mistakes, so I can learn more. I have tried to not make mistakes. I found Will Gompertz’s analysis of the situation useful, and he notes the basic breakdown of the COVID arts and culture grant we know today:

The £1.15bn support pot for cultural organisations in England is made up of £880m in grants and £270m of repayable loans. The government said the loans would be “issued on generous terms”.

Funding will also go to the devolved administrations – £33m to Northern Ireland, £97m to Scotland and £59m to Wales.

A further £100m will be earmarked for national cultural institutions in England and the English Heritage Trust.

There will also be £120m to restart construction on cultural infrastructure and for heritage construction projects in England that were paused due to the pandemic.

The government said decisions on who will get the funding would be made “alongside expert independent figures from the sector”.

Coronavirus: Emergency money for culture ‘won’t save every job’ 7 July 2020

I am definitely glad to see that the cultural sector has been recognised as having value and need for support. This is unequivocally good. The very early point I am trying to make is that there might be a way to look past and around and through the giant nationals with the loudest voices and ongoing DCMS support in the millions and with vast assets (many of whom as speaking to us via that 5 July government press release to say how happy they are) to see if it’s possible, finally, to illuminate the smaller players, the dynamic and struggling groups, the covens of freelance talent, the support companies, and basically everyone else who isn’t one of the biggies.

Staring into the status quo

I chatted about this with a few arts-related friends, and Clare directed me to a report called The Art of Dying written in 2005 by John Knell. I hope everyone who’s dispensing funds has studied it and can recite it from start to finish. It’s a response to a conference held the year before, where these three main insights were born, and I quote:

1. That the portfolio of arts organisations in the UK has become too fixed

2. That there are too many undercapitalised arts organisations, operating at near breaking point organisationally and financially, whose main preoccupation is survival diverting their energies from the central mission of cultural creativity

3. That we need to provoke a more challenging public conversation about the infrastructure supporting the arts in the UK, and the strategy and modus operandi of arts organisation

The Art of Dying

I really like what Mr. Knell is writing in this paper – it’s definitely worth your time to read it. It’s important to be able to look at each other and agree that an organisation with £1,001,693,000 worth of assets is stable. Or bloody well should be.

So interesting. 

Chatting further with more arts and culture colleagues, I was encouraged — thanks, Fiona — to reframe the question to: Who Needs It The Most? This is a much harder question. I’d consider myself to be a true friend to all museums everywhere, but I have to admit I particularly love the small ones that are super fucked, and definitely don’t have £1,001,693,000 hiding away in real estate or other investments that are difficult to access because there’s some form of governance in the way of deciding to release them. 

As I look at the big, open, reported numbers, I will also be on the hunt for the numbers hiding in plain sight, or not documented at all. And please, if you can direct me to good reporting on arts and culture networks and their funding, I would absolutely love the steer.

V&A DIRECTOR OF RESEARCH AND COLLECTIONS JOINS GOOD, FORM & SPECTACLE BOARD

We’re beyond thrilled to welcome Professor Bill Sherman to the board of Good, Form & Spectacle. The fact that Bill is Director of Research and Collections at the Victoria & Albert Museum makes all this even better.

sherman3Bill Sherman is Director of Research and Collections at the Victoria and Albert Museum, where he is leading the development of a V&A Research Institute. He moved to the V&A from the University of York, where he was Director of the Centre for Renaissance & Early Modern Studies. He works on the ways in which objects come down to us from the past, what they pick up along the way, and how they speak across time and space. He is best known for his work on marginalia and has published widely on the history of reading and the plays of Shakespeare and his contemporaries.

Our eyes first met across a crowded project called “History of an Object in 100 Worlds”, which I’d discovered in my own research around The Small Museum idea. The project’s brilliant, and back in September 2014, I left a note on its blog saying hello and could I please learn more. From there, I went to meet with the project’s leader, Lina Hakim, who introduced me to Bill.

We have a shared interest in objects’ histories. The History of an Object pilot project resulted in a short series of V&A staff presenting their own stories about favourite objects in the museum. It’s a great read, and I hope it gets circulated more widely. Having met quite a few subject experts by now, it’s one of my favourite, favourite things to have the privilege to listen to them tell me about the objects they study.

But, back to Bill. He’s already been hugely helpful to me, both as an advisor and know-er of most of the London cultural sector. He’s also largely responsible for the successful instigation of the new V&A Research Institute (VARI), “a new programme of research and teaching partnerships that will enhance access to the V&A’s collections and develop new approaches to research, training, display and interpretation.” Given that G,F&S is also heavily invested in R&D, it just seems like a great fit.

Bill says, “George and G,F&S are full of great ideas for making collections more accessible and more interesting to a wider range of people–inside and outside the walls of museums.”

Watch this space! Welcome, Bill.

Museum in a Box V2

Our Museum in a Box project is moving forward. I’ve spent the day making two new boxes, one 3D, the other 2D. These are just to think with really, and show people, as we continue to flesh things out. It’s about exploring something like object-oriented experience design.

  
    
 

We’ve set up another Twitter account for just this project, @_museuminabox. It’ll need a proper web presence soon too, so there a coming soon page online as we work on it at museuminabox.org.

You can read about V1 of the box here.

Internal R&D Project #4: Two Way Street

It’s pretty late on a Friday afternoon, possibly the dumbest time to launch something, but, my conspirators and I decided to only work on this this week, so, we kind of have to launch it now.

The thing is called Two Way Street, and it’s a new way to explore The British Museum collection. It’s truly a museum of the world for the world, and we think Two Way Street is fantastic for looking around. Our team was George Oates, Tom Armitage, Frankie Roberto, with a cameo data-munging appearance by computer scientist, Tom Stuart. Thanks also to Harriet Maxwell and Tom Flynn for working on the (unsuccessful) proposal to NESTA for funding, Felix Ostrowski for RDF-to-JSON advice, and Barry Norton for restarting the BM SPARQL endpoint.

Two Way Street is basically an exploded view of the catalogue. Once we’d processed the big catalogue into a format that was easier for us to work with, we built just a few simple template views on top of the catalogue. We also skewed the user experience towards learning about the acquisition history of the museum. There are some really interesting trends and people involved in the formation of the institution. The British Museum was founded in 1753, and is the world’s first public museum.

Here’s the home page, where we introduce the first of a handful of visualisations, acquisitions over time, by decade.

chloe

We’re also able to display a bunch of facets we selected as interesting. You can use them as leaping-off points into the collection. There’s another subtle visualisation there to show you which facets are well-understood in the metadata.

thing

Here, you can see a list of all the people (or institutions) who found, excavated, or collected things…

facets

Like Chloe Sayer, who found/excavated/collected 6,296 things in the later decades of the Twentieth century…

found

It’s a Ruby, Elastic Search, Heroku, AWS-y thing. We’re also making use of the British Museum’s data dump from last August, and hitting their SPARQL endpoint (possibly a bit harder than everyone is used to). I like to think we’re some kind of “cultural white hats” that might actually be able to constructively help the museum to understand and develop the infrastructure it needs to serve more external development.

There’s a little more about it all on the site’s About page, if you’d like to go and have a look. Tom’s going to follow up, too, with some thoughts on using Elasticsearch instead of a database, which we all through was pretty cool.

Version 1: Day 1

It’s possible that the post title implies that I’ll be posting every day of our ten-day Small Museum stint at Somerset House, but it doesn’t. 

It was, however, a CRACKING DAY ONE. Harriet, Tom and I are hanging out at thesmallmuseum.org doin’ stuff.



https://flickr.com/photos/34427469121@N01/sets/72157650971527057

Internal R&D Project #3: Museum in a Box

We have exciting news on several fronts today.

First, friend-of-GF&S Cassie Robinson has offered us a pop-up space at Somerset House for two weeks, starting Wednesday. It’ll be next to Cassie’s excellent Civic Shop. We’ve decided to prototype a couple of big ideas in public there.

The Small Museum

The Small Museum is a major research project we’re undertaking with the basic provocation of What could a 21st Century museum be like if you built it from scratch? We’re going to start our very first physical instantiation of this question at Somerset House.

For ease of use, we’ve started a new, separate website to document our work: thesmallmuseum.org. We’ll also be tweeting from @thesmallmuseum, if you’d like to follow along.

Museum in a Box

Last Friday, we received our prototype set of objects for our third internal R&D project, called Museum in a Box.

goddess

The prototype box contains 10 objects from The British Museum’s collection. We’re working on building a page for each of them now. The short term plan for our next two weeks is to prototype ideas around the box, content for each object, and a handful of different interaction design models for engagement. It suddenly dawned on me last week that our short stay at Somerset House is actually a fantastic environment to talk to people who are visiting semi-randomly about the idea and see what they think. I’ll be joined by Harriet Maxwell and Tom Flynn.

We’re expecting a bunch of visitors by appointment too, so if you’d like to come and hang out (and ideally participate and even make something with us), we’d love it.

There’s already a bit of chatter about the project in its very early days…

https://twitter.com/antimega/status/576060295170433024

And, just last Saturday, I took the prototype — actually a Museum in a Bag for now — to a fun collaboration/pitching day at Ideas Create in Kent. It was really enjoyable and useful to hear what people thought about the general idea. (Thanks to Jon Pratty for the invitation to participate too.)

Sketching and Engineering

This is a guest post from Tom Armitage, our collaborator on the V&A Spelunker. It’s our second internal R&D project, and we released it last week.


Early on in the process of making the V&A Spelunker – almost a few hours in – I said to George something along the lines of “I’m really trying to focus on sketching and not engineering right now“. We ended up discussing that comment at some length, and it’s sat with me throughout the project. And it’s what I wanted to think about a little now that the Spelunker is live.

For me, the first phase of any data-related project is material exploration: exploring the dataset, finding out what’s inside it, what it affords, and what it hints at. That exploration isn’t just analytical, though: we also explore the material by sketching with it, and seeing what it can do.

The V&A Spelunker is an exploration of a dataset, but it’s also very much a sketch – or a set of sketches – to see what playing with it feels like: not just an analytical understanding of the data, but also a playful exploration of what interacting with it might be like.

Sketching is about flexibility and a lack of friction. The goal is to get thoughts into the world, to explore them, to see what ideas your hand throws up autonomously. Everything that impedes that makes the sketching less effective. Similarly, everything that makes it hard to change your mind also makes it less effective. It’s why, on paper, we so often sketch with a pencil: it’s easy to rub out and change our mind with, and it also (ideally) glides easily, giving us a range of expression and tone. On paper, we move towards ink or computer-based design as our ideas become more permanent, more locked. Those techniques are often slower to change our minds about, but they’re more robust – they can be reproduced, tweaked, published.

Code is a little different: with code, we sketch in the final medium. The sketch is code, and what we eventually produce – a final iteration, or a production product – will also be code.

As such, it’s hard to balance two ways of working with the same material. Working on the Spelunker, I had to work hard to fight the battle against premature optimisation. Donald Knuth famously described premature optimisation as ‘the root of all evil‘. I’m not sure I’d go that far, but it’s definitely an easy put to fall into when sketching in code.

The question I end up having to answer a lot is: “when is the right time to optimise?” Some days, even in a sketch, optimisation is the right way to go. If we want to find out how many jumpers there are in the collection – well, that’s just a single COUNT query; it doesn’t matter if it’s a little slow.

I have to be doubly careful of premature optimisation when collaborating, and particularly sketching, and remember that not every question or comment is a feature request. My brain often runs off of its own accord, wondering whether I should write a large chunk of code, when really, the designer in me should be just thinking about answering that question. The software-developer part of my brain ought to kick in later, when the same question has come up a few times, or when it turns out the page to answer that question is going to see regular use.

For instance, the Date Graph is also where the performance trade-offs of the Spelunker are most obvious. By which I mean: it’s quite slow.

Why is it slow?

I’d ingested the database we’d been supplied as-is, and just built code on top of it. I stored it in a MySQL database simply because we’d been given a MySQL dump. I made absolutely no decisions: I just wanted to get to data we could explore as fast as possible.

All the V&A’s catalogue data – the exact dataset we had in the MySQL dump – is also available through their excellent public API. The API returns nicely structured JSON, too, making an object’s relationships to attributes like what it’s made of really clear. A lot of this information wasn’t readily available in the MySQL database. The materials relations, for instance, had been reduced to a single comma-separated field – rather than the one-to-many relationship to another table that would have perhaps made more sense.

I could have queried the API to get the shape of the relationships – and if we were building a product focused around looking up a limited number of objects at a time, the API would have been a great way to build on it. But to begin with, we were interested in the shape of the entire catalogue, the birds’ eye view. The bottleneck in using the API for this would be the 1.1 million HTTP requests – one for each item; we’d be limited by the speed of our network connection, and perhaps even get throttled by the API endpoint. Having a list of the items already, in a single place – even if it was a little less rich – was going to be the easiest way to explore the whole dataset.

The MySQL database would be fine to start sketching with, even if it wasn’t as rich as the structured JSON. It was also a little slow due to the size of some of the fields – because the materials and other facets were serialized into single fields, they were often quite large field types such as LONGTEXT, which were slow to query against. Fine for sketching, but it’s not necessarily very good for production in the long-term – and were I to work further on this dataset, I think I’d buckle and either use the API data, or request a richer dump from the original source.

I ended up doing just enough denormalizing to speed up some of the facets, but that was about it in terms of performance optimisation. It hadn’t seemed worthwhile to optimize the database at that point until I knew the sort of questions we want answered.

That last sentence, really, is a better answer to the question of why it is slow.

Yes, technically, it’s because the database schema isn’t quite right yet, or because there’s a better storage platform for that shape of data.

But really, the Spelunker’s mainly slow because it began as a tool to think with, a tool to generate questions. Speed wasn’t our focus on day one of this tiny project. I focused on getting to something that’d lead to more interesting questions rather than something that was quick. We had to speed it up both for our own sanity, and so that it wouldn’t croak when we showed anybody else – both of which are good reasons to optimise.

The point the Spelunker is right now turns out to be where those two things were in fine balance. We’ve got a great tool for thinking and exploring the catalogue, and it’s thrown up exactly the sort of questions we hoped it would. We’ve also begun to hit the limits of what the sketch can do without a bit more ground work: a bit more of the engineering mindset, moving to code that resembles ink rather than pencil.

“Spelunker” suggest a caving metaphor: exploring naturally occurring holes. Perhaps mining is a better metaphor, and the balance that needs to be struck digging your own hole in the ground. The exploration, the digging, is exciting, and for a while, you can get away without supporting the hole. And then, not too early, and ideally not too late, you need to swap into another other mode: propping up the hole you’ve dug. Doing the engineering necessary to make the hole robust – and to enable future exploration. It’s a challenge to do both, but by the end, I think we struck a reasonable balance in the process of making the V&A Spelunker.

If you’re an institution thinking about making your catalogue available publicly:

  • API access and data dumps are both useful to developers depending on the type of work they’re doing. Data dumps are great for getting a big picture. They can vastly reduce traffic against your API. But a rich API is useful for integrating into existing platforms, especially if they make relatively few queries per page against your API (and if you have a suitable caching strategy in place). For instance, an API is the ideal interface great for supplying data about a few objects to a single page somewhere else on the internet (such as a newspaper article, or an encyclopedia page).
  • If you are going to supply flat dumps, do make sure those files are as rich as the API. Try not to flatten structure or relationships that’s contained in the catalogue. That’s not just to help developers write performant software faster; it’s also to help them come to an understanding of the catalogue’s shape.
  • Also, do use the formats of your flat dump files appropriately. Make sure JSON objects are of the right type, rather than just lots of string; use XML attributes as well as element text. If you’re going to supply raw data dumps from, say, an SQL database, make sure that table relations are preserved and suitable indexes already supplied – this might not be what your cataloguing tool automatically generates!
  • Make sure to use as many non-proprietary formats as possible. A particular database’s form of SQL is nice for developers who use that software, but most developers will be at least as happy slurping JSON/CSV/XML into their own data store of choice. You might not be saving them time by supplying a more complex format, and you’ll reach a wider potential audience with more generic formats.
  • Don’t assume that CSV is irrelevant. Although it’s not as rich or immediately useful as structured data, it’s easily manipulable by non-technical members of a team in tools such as Excel or OpenRefine. It’s also a really good first port of call for just seeing what’s supplied. If you are going to supply CSV, splitting your catalogue into many smaller files is much preferable to a single, hundreds-of-megabytes file.
  • “Explorer” type interfaces are also a helpful way for a developer to learn more about the dataset before downloading it and spinning up their own code. The V&A Query Builder, for instance, already gives a developer a feel for the shape of the data, what building queries looks like, and clicking through to the full data for a single object.
  • Documentation is always welcome, however good your data and API! In particular, explaining domain-specific terms – be they specific to your own institution, or to your cataloguing platform – is incredibly helpful; not all developers have expert knowledge of the cultural sector.
  • Have a way for developers to contact somebody who knows about the public catalogue data. This isn’t just for troubleshooting; it’s also so they can show you what they’re up to. Making your catalogue available should be a net benefit to you, and making sure you have ways to capitalize on that is important!