My app review experiences

posted 2020-11-08, filed under "apple antitrust"

Hi, I’m Theodore. I created iSH, the Linux terminal for iOS, and I’ve led its development since then. Apple recently approved our app, then told us they were going to take down our app only 3 business days later. They gave us 2 weeks, which we’ve spent talking to App Review and trying to understand what we can do about it. This whole process has been a lot more difficult and stressful than it needs to be, and ultimately didn’t get us much closer to understanding what’s wrong.

Unwritten Rules

The story starts in May, when we first tried submitting iSH for review. We checked the App Store Review Guidelines and it seemed like iSH would be compliant. But it was rejected under guideline 2.5.2, saying “Specifically, your app allows the user to install linux executable code..” It seems like while we can type anything we want in an App Store Connect message, reviewers can only choose from a list of canned responses, which don’t have a lot of room to say arbitrary things. Since this was a canned response, we asked for clarification and were told that they would call us at some unspecified time within the next 3-5 days. This is the first problem with App Review: if they set up a call, they don’t seem to ever tell you when that call will be, only a vague range. And the phone call could have just been an email: after an anxious 2 day wait, someone called us to read a prepared statement saying that what’s not allowed is “package management or loading code of any kind from local or remote sources.”

iOS developers know that the guidelines are merely suggestions, and you can only really find out the real rules by submitting an app for review, getting a rejection, and then asking for clarification. We had guessed Apple might flag the app under 2.5.2, but this confirmed that there’s an unwritten subrule of 2.5.2 covering package management functionality.

We Can’t Pre-Approve Your App

I tried to get a little more info at WWDC. After COVID-19, Apple revamped the event to be entirely remote, and as part of this made their labs (events where you can talk to Apple employees in person) available to anyone over the Internet without having to win a lottery and buy a $1600 ticket. (I really hope they can keep doing this after the pandemic is over.) One of the topics is App Review, so I signed up for a session. I talked to someone I’ll call Bob, who took some time to explain the spirit of 2.5.2: “What we don’t want is an app that’s able to change its functionality after approval.” He listed some types of apps that don’t follow the spirit: “app previewers” (there are apps in the store whose purpose is evaluating arbitrary React Native code you write on your computer) and “remote package importing” (there are popular apps on the store that run Python code and include a GUI for pip). I asked whether removing apk would be enough, and he gave what I’ve learned is the default non-answer of App Review: “we can’t pre-approve your app, but submit it and see what happens.” So that’s what we did.

Actually updating the app to remove apk took a while. We submitted this on October 20, it was approved the next day, and we launched on the 22nd. The celebration was short-lived though: Apple called us without warning on the 26th. Normally when Apple calls, they first send you a message in App Store Connect and read you the same exact message over the phone. This time they reversed the usual order and called first. The call came from someone I’ll call Mike, who told us that “wget” is also a form of package management. This was followed by the exact same message in App Store Connect.

Remote Code Importing

Soon after we released iSH without apk, we found users who had figured out a way to get apk back using wget and were posting about this throughout the internet. We suspect this was the reason why our rejection notice mentioned wget specifically. This made it clear that App Review was just trying to enforce the unwritten rule we’d found earlier, but without a basic understanding of what wget actually is. We never told users how to get apk, but they figured it out anyway. The nature of iSH meant that this problem was fundamental, as users can always add back functionality that we remove. They could figure out a way to add back wget, just like they were able to so easily add back apk.

Of course we tried to get some clarification on this. I pointed out that wget just downloads files, and asked why Safari is allowed to download files while wget is not. The answer I got was that this is because Safari is a browser and iSH is a terminal UI, which seems nonsensical to me. Mike said he wasn’t very technical and didn’t understand much about this topic, so he had to ask the “tech folks” behind the curtain. I’m guessing Mike’s entire job is calling people, not reviewing apps. It certainly doesn’t seem like has any say in our case. I suspect the tech folks he was checking with are a team of technically inclined reviewers (or even just one) who actually review the app and write the rejection messages we’ve been seeing. It doesn’t appear to be possible to speak directly to these people, and instead we have to pass messages through Mike, like in a game of telephone.

Since Mike kept talking about “remote code importing,” I asked if local code importing would be a different situation. The answer he got was interesting: any kind of code importing would not be appropriate because it would create an alternative App Store.

We wrote up a long message with all of our concerns, and they got back and told us they’d call us in the next 24-48 hours. The wait was about as stressful as waiting for Nevada to release their election results. Finally Mike called again, and he started the call by just reading back the same rejection message we’d heard before, without really addressing our concerns. He did go into more detail on what remote package importing means: It’s OK to execute code that you type in the app, but not code obtained from external sources. This doesn’t seem very well thought out: nothing can stop you from copying and pasting code from external sources into the app. When asked specifically “are you saying copy/paste is OK while drag/drop is not,” he asked the tech folks who declined to answer (“we can’t pre-approve anything”). He also brought up a bizarre-sounding “core concern” that that a Linux terminal on iOS is a security risk. It’s not clear how, but it sounds like a “attacker can execute malicious code on their own machine” sort of vulnerability.

Appeals

At this point we realized that their interpretation of the guidelines didn’t make much sense, and tried suggesting a change to the guidelines. This was a new appeal option they had added back in June after the Hey fiasco. I asked Bob about this at WWDC. He said he didn’t know any more than me about how guideline appeals would work, but that the existing App Review Board was the same people who maintain the guidelines, and already had the power to change guidelines in response to an appeal. To us, it seems likely that nothing has actually changed, and it was a sort of token announcement with no real effect. I have yet to see any evidence of any suggested guideline change being seriously considered, including ours.

By this time, it had already been a week since the original message, leaving just one more week before iSH would be removed from the store. With time running out, we tried submitting an App Review Board appeal, but heard nothing back—not even a message saying they were looking into it and would extend the deadline. With no other options, on Thursday we emailed Phil Schiller. Unsurprisingly, we didn’t get anything out of that either—leaving us where we are today.

We’ve spent these past two weeks trying to get the situation with iSH resolved, but even with all our effort we’re no closer to a solution. Instead, I and the other iSH contributors have had an incredibly stressful fourteen days as we tried to bend over backwards to meet Apple’s arbitrary scheduling while also juggling our full-time jobs. Feature development on iSH has all but stopped—we certainly haven’t had time to work on it while wrestling with App Review, and external contributions have all but dried up under the spectre of it being “wasted” if Apple rejects the app. And while everyone we’ve been able to contact has done their best to be helpful, the App Review process makes it fundamentally difficult for us to receive the information we need to ensure iSH can meet Apple’s requirements. Its various provisions for indicating errors or misunderstandings on the part of the App Review team seem to have not done much of anything in our case.

It doesn’t have to be this way. Many of the things we had issues with would be trivial for Apple to fix. Simply preferring emails rather than unscheduled phone calls would go a long way in making our lives easier, as would pausing enforcement of the deadline during the appeal process. Having people on hand who can answer our questions beyond what’s written on a sheet of paper would have done wonders for our compliance efforts. As it stands, we’ve spent two weeks doing Apple’s job for them with what has been essentially zero results. The process seems to have been designed to maximize convenience to Apple without much regard to how inconvenient this is for developers. Apple can and should do better.