Dear Computer Architecture Community,
This blog is to provide an overview of the HPCA 2021 PC organization and the paper selection process. As you all know by now, HPCA 2021 will be virtual. So, let me take this opportunity to invite you all to the first-ever virtual and the 27th edition of the IEEE International Symposium on High-Performance Computer Architecture (HPCA-27). The conference was originally scheduled to be held in Seoul, South Korea. Due to COVID-19, it is being held virtually. It is my honor to chair the technical program committee and work with so many brilliant minds who served on the program committee (PC) and the external review committee (ERC). The many hours/days/weeks spent voluntarily reviewing the papers by the PC and ERC members is a testament to our community’s collective dedication to showcasing the best of computer architecture research to the world. What is fascinating about our community is its agility to adapt to shifts in the technology landscape and the market forces that drive computing to new use case scenarios. This adaptation is evident in technologies such as quantum computing which will have more than one dedicated session at HPCA-27.
Technical Program Overview:
HPCA-27 will present 63 technical papers (probably the largest ever HPCA) on a wide range of topics from accelerators, security, storage and memory systems, data center design, performance modeling, quantum computing, and more. I am also pleased to announce that Dr. Nam Sung Kim, Senior Vice President of Samsung Electronics, and Professor at the University of Illinois at Urbana Champaign, will be the keynote speaker. Dr. Kim has been the force behind the introduction of processing in-memory systems at Samsung, which aims to accelerate many data intensive application domains. The industry track program selection was lead by Dr. Gilles Pokam, and Prof. Dan Sorin lead the selection of the papers for the “Best of CAL” session.
Joint Session Panel on Diversity, Equity, and Inclusion:
We are also pleased to be able to bring a joint panel titled “Valuing Diversity, Equity, and Inclusion in Our Computing Community,” which is being jointly organized across all the co-located conferences with HPCA, namely PPoPP, CGO, and CC. Timothy Pinkston is organizing the panel with an outstanding roster of panelists from our community concerned about this issue. He plans to write a separate blog about the panel which can help contextualize related issues and efforts. I personally believe the topics of the panel are of significant interest and importance to our entire community, irrespective of our geographic and work affiliations. To make this panel broadly accessible, anyone can register for and attend the panel for free. The free registration is available at the following link: https://whova.com/portal/registration/ppopp_202102/. We hope to attract participants from across the wide spectrum of the computing community. So please send this link to any of your colleagues, students, or anyone else who you think could be interested in this important issue and encourage them to attend this free panel. We hope this open dialogue will not be a one-time event but will be repeated frequently and widely, and more critically, we hope it will lead to lasting changes to the benefit of our computing community.
Technical Program Selection Process:
Coming to the technical program selection, I had the advantage of a whole year ahead to plan for the program selection process since I was invited to chair the TPC nearly 15 months before the conference. I exploited that advantage by sending out the PC invitations as early as February 4th, 2020. I am grateful to the 90 PC members and the 86 ERC members who agreed to serve on the PC. The PC meeting was split across two days where we had two mostly disjoint sets of PC members attending each day and few PC members who served on both days. It took me 10 days to go through about 260 submitted papers to verify conflict of interests and reading abstracts to find appropriate review assignments. The conflict of interest verification process was tedious and slow. While I used the double-blind review process it was difficult to assure anonymity in the presence of Arxiv and other fast-growing public disclosure mechanisms. Nonetheless, I provided some basic guidance to authors to protect their paper’s anonymity without impeding an author’s ability to put their work on public disclosure sites.
The PC members were each assigned about 13 papers and the ERC members were mostly assigned 3 papers and in some rare cases up to 5 papers. Each paper was assigned 5 reviewers at least, with 4 reviewers from the PC pool and one reviewer from the ERC pool. I used a single round review process to provide equal opportunity to all submitted papers. The reviewers had about 6 weeks to finish the reviews. The authors were given two weeks to provide either a rebuttal or a revised paper to address reviewer comments. At the end of the rebuttal window, the PC and ERC members followed a rigorous online deliberation to categorize the papers. About half the papers were marked for discussion during the PC meeting and a few of them were accepted online.
The two-day PC meeting (Oct 23-24th 2020) which was originally scheduled to be held in Los Angeles had to be moved to a virtual PC meeting. We started the PC meeting at 6AM Pacific Time each day and the meeting ran for 8-10 hours each day. I ranked all the papers based on the average post rebuttal score for creating a discussion order. However, due to the virtual nature of the PC meeting I had to organize paper discussions to meet the time zone limitations of PC members. I wanted to be cognizant of my colleagues in Asia and Europe with late night times. As such, I assigned paper discussions to go slightly out of order to enable all reviewers to participate in the meeting. But it turned out that our community is well versed with all-nighters and most PC members were present for nearly the entire meeting! Special thanks to all the PC members from Asia and Europe who had to do all-nighters to accommodate the virtual setting.
Each discussed paper was assigned a discussion lead who summarized the paper and presented his/her view. The discussion lead also accounted for the ERC review. It was followed by comments from other reviewers. Each paper was discussed for a minimum of 6 minutes (many more minutes in reality). This discussion was followed by a vote amongst the PC reviewers only. The majority decision from the paper reviewers was used for paper selection. Since each paper had 4 PC reviewers assigned, I felt comfortable using the majority rule. Only in case of a tie we had an additional round of discussion followed by a vote of the entire PC. I believe only a handful of papers fell in this category.
The vast majority of the accepted papers had a shepherd assigned to provide personalized guidance to the authors. To keep the shepherding process fair, I adhered to the double-blind shepherding process. The shepherding process was smoother than I anticipated and only in a few cases, I had to act as an intermediary to help communicate the shepherd’s requests to the authors and vice-versa.
Lessons Learned:
Conflict management: The challenge of verifying conflict of interests is the most difficult one to overcome for a PC chair without authors voluntarily and strictly adhering to the honor code. I had to make the unfortunate decision of desk rejecting one submission for a clear violation of the honor code, after consulting the TCCA executive committee. I had to also send several clarification emails to get the conflict information. As the size of the PC and ERC grows in the future the manual selection of conflicts on HotCRP is also a tedious task for the authors. I believe we need a centralized database that maintains conflict lists, including the type of conflict and the duration that will make future PC chair jobs more manageable.
Time to rethink double-blind: The notion of the double-blind review process, in my view, is now past its prime. We must figure out a better way to handle Arxiv submissions. Maybe time for a single-blind review process? For a reviewer who has the time and interest to find out the authors, it seems they are only a few clicks away. But more critically even searching for related prior work can potentially reveal author names. It is best to tackle this issue head-on rather than avoiding the issue of Arxiv. I provided some basic guidelines to reviewers to not search for the title or key phrases in the paper. I also provided guidelines to authors to use a different title from Arxiv and minimize the use of terms that may link to their Arxiv submission. I don’t know how many followed this guidance from either end. With the increasing popularity of Arxiv, I believe soon there will be no double-blind process in practice — I think the playing field will be level at that time.
Virtual-only PC Meetings: I believe virtual PC meetings are the way to go in the future. It reduces unnecessary travel miles, fatigue, and also enables a broader group of community members from different parts of the world to participate in the review process. I think it is time for our community to consider virtual PC meetings as the norm going forward. However, we do need a mechanism to allow our young research colleagues to interact with the PC members in this virtual setting. Maybe setting up a time window (a couple of hours) to host a social hour or a research poster session will help remedy this side effect of virtual PC meeting.
Revisions: Providing an opportunity for all the submissions to submit a revision turned out to be not a good option in hindsight. I think inviting only a select few papers to submit revision explicitly would be a better use of reviewer time (and authors’ time as well). The reviewers can indicate whether they want to see the revision to make such a choice.
Note of gratitude:
- Chairing the TPC without a multitude of helping hands is beyond my imagination. I am indebted to my Ph.D. research group for providing those helping hands. Haipeng Zha and Yongqin Wang were instrumental in maintaining the PC invitation status, testing the HotCRP submission site, creating various scripts for virtual PC management through Zoom. My students Abdulla Alshabanah, Keshav Balasubramanian, Hanieh Hashemi, Krishna Giri Narra, Rachit Rajat, Tingting Tang diligently acted as scribes in capturing the main discussion points for each paper during the PC meeting.
- Milos Prvulovic handled all those papers where I had a conflict. He was immaculate in his handling of this responsibility.
- Mattan Erez and Jun Yang provided valuable HotCRP guidance and Zoom management scripts from their MICRO PC meeting experience. Lieven Eeckhout provided guidance on the review of ethics policies.
- Josep Torellas, David Kaeli, Daniel Jimenez, Lizy John, and Antonio Ortega were my go-to source for any issue related to the HPCA TPC process. They were generous with their time and guidance on many thorny matters.
- Eddie Kohler at HotCRP was my key to understanding the HotCRP pandora box.
- Junh Ho Ahn and John Kim are superb in their general chair organization; I have a special appreciation for their trials and tribulations from my ISCA 2018 general co-chairing responsibility.
- Thanks also to the PC and ERC members who volunteered all their hours to make my responsibility a pleasurable one.
- To the authors of all the submitted papers, irrespective of the paper outcome, I hope you found the feedback valuable. Without all of you, there will be none of this!
I hope you all enjoy this first ever virtual HPCA and make use of the research results in pushing the computer architecture envelope farther.
Murali Annavaram,
Professor,
University of Southern California