Mountaintop Madness – NOV 19/20

The past eight months have been a busy time for the PAPA technical team. A relentless series of winter storms made the higher elevation sites impossible to service due to ice and snow accumulation throughout the winter and most of the spring. As a result, the Sunset Ridge and Toro Peak repeaters were off the network for some time. While these two sites required urgent restoration, equipment at San Marcos Peak, Mt. Woodson, Pleasants Peak, Elsinore Peak, Santiago Peak, Blueridge, Mt. Wilson, and Oat Mountain benefited from pre-planned upgrades and regular preventative maintenance activities.

Problems at Toro Peak began in late November of 2019, when Cecil, WD6FZA, and I made a regular maintenance trip to Edom Hill. On the drive back we noticed that the power output on the Toro Peak analog repeater appeared to be down considerably. While we were able to get into the receiver just fine, it was difficult to hear the transmitter in areas that were typically saturated with coverage. The site at Toro Peak (P07) is three hours away from Cecil’s driveway, so a trip to the hill takes all day. It’s impossible to be over prepared for a trip to Toro Peak. Not knowing what the issue might be with the low output power, we strategized on what to bring with us on a repair run.

At the top of the list was a replacement RF power amplifier. Because of the high duty cycle on the PAPA analog system, it’s not unusual to have one or two amplifier failures per year across the sites. With no spares in stock, Cecil called Ted Henry and a new UHF 100W PA was assembled and tested in two days. We had the new PA by the following Monday. We scheduled a trip for the very next day which was only two days before Thanksgiving. Tuesday morning at 0600, we loaded the Jeep with the new PA, a replacement repeater (in case the repeater was the problem), and a full array of test equipment and set out for the site.

The drive to the top takes nearly two hours from the gate off the highway. The lower section of the road was in good condition and there didn’t appear to be a lot of snow accumulation at the top of the mountain. However, as we began to climb on the northwest side of the mountain, things began to look worse. This side of Toro Peak is shaded from the sun most of the day so the snow only melts late in the afternoon and then quickly freezes again after sunset. This condition results in dangerous ice formations on the road. On a mountain road only wide enough for a single vehicle with steep drops off either side, ice is not something you want to encounter.

About 50 minutes into the ascent, we sat motionless on a sheet of ice with no traction. I was able to slowly back up about 15 yards until the Jeep was controllable again. I continued backwards down the hill until I found a spot wide enough to allow a multi-point turn to get pointed down the hill. Mother Nature put an end to the repeater rescue trip on this attempt.

The first week of December saw the installation of a new controller at San Marcos Peak (P09). New controllers are being installed across the system to improve our ability to control remote sites and standardize hardware for greater efficiency during maintenance trips. The installation went well and the new controller has been working fine since. Another trip was made to the site the following week to work on the IRLP computer. A problem in the BIOS setup prevented the computer from automatically starting following a power failure. In addition, an intermittent power connector would result in the occasional failure of the node. The problems were remedied on this trip.

Later that same week, a trip was made to Mt. Woodson (D15) to replace a failed D-STAR gateway computer. Ira, KI6TPX, pre-configured the computer and it was easily installed at the site. Soon after, the D-STAR repeater was working normally.

With Toro Peak inaccessible, and the remainder of the PAPA System working well, things quieted down for the holidays. In late February, another trip was made to Mt. Woodson to move the site hardware from the six foot Motorola cabinet that housed it into into a new seven foot server cabinet. The server cabinet was needed to provide the extra depth required to accommodate the abnormally deep D-STAR repeater and provide room for expansion and/or upgrades in the future. After a few hours of work at the site everything was transferred over and tested.

In what looked like a lull between winter storms, we planned a trip to get to Toro Peak in the second week of March. This time, when we arrived at the gate off the highway, we found it locked in an attempt to keep campers off the mountain during the winter conditions. The key that we have used in the past to open the gate would not work on any of the locks. We drove back to the Santa Rosa Indian Reservation to see if we could obtain a new key to the gate. The Toro Peak site is owned and managed by the Santa Rosa Band of Cahuilla Indians. At the Reservation, we were told that the US Forest Service had changed the lock and had not yet provided a new key for distribution to the site tenants. We were informed that the Band had its own lock on the gate and that they would unlock the gate for us.

After losing an hour off our schedule, we arrived back at the gate with a representative from the Band. Apparently, their lock had not been used in some time and was fused shut. After about 30 minutes of trying, it was obvious that we were to be thwarted again in our attempt to restore service on Toro Peak. For the second time, we turned around and drove home.

The following week we made a trip to Sunset Ridge to investigate the failure of the 220 repeater (P13). There was still snow on the hill but the outdoor camera at the site gave us some hope that the trip would be possible. It wasn’t to be. After arriving at the gate at the trail-head, we found that the combination lock had been changed and we couldn’t open the gate. We drove back to the US Forest Service office in Mt. Baldy. The Ranger loaned us a key to open the gate and we drove back to the trail-head. The loaner key wouldn’t work on any of the locks on the gate so we again drove back to Mt. Baldy to return the key. We drove back to San Marcos without being able to access the site.

In late April, I made a visit to Elsinore Peak (P24) to install a new repeater controller. Typically, when site upgrades are planned, a replacement repeater is constructed in the PAPA test rack and burned-in for a period of two weeks. Since the majority of the site hardware is harmonized across the system, it’s easier to build a duplicate repeater package and simply replace it on the hilltop for the prior one. The repeater system that is replaced is then installed in the test rack for rotation to the next site. This saves time at the sites and helps eliminate surprises that can stop down an installation.

The repeater that was replaced at Elsinore Peak was put in the test rack the next day and paired with another new repeater controller. After just a week of burn-in (the site hardware was already proven at Elsinore Peak), the new repeater package was taken to Pleasants Peak (P22) for installation. The trip to Pleasants Peak was successful and two PAPA analog sites were upgraded in the period of a single week. This was welcome success after the difficulties in getting to the Sunset Ridge and Toro Peak sites in the weeks prior.

In preparation for a return visit to Sunset Ridge, we determined that we should bring with us the backup repeater in case we found a repeater failure that could not be resolved on the site. The backup 220 repeater, however, was located at Thousand Oaks (P25) because the primary repeater at that site had failed months earlier. In addition, a co-channel conflict with a repeater outside of Bakersfield surfaced that was best solved with a channel change on the Thousand Oaks UHF repeater (P26). So, in the middle of the COVID-19 lock-down, Cecil and I made the trip to Thousand Oaks to install the repaired 220 repeater and re-channel the UHF repeater.

The drive up the 405 freeway during the first week in May was like nothing I’ve seen before. We made it from Oceanside to the 23 freeway in under two and one-half hours! On a normal traffic day, we’d be lucky to be at LAX in that amount of time. Our freeway speed never dropped below 70 mph. That was until we made the transition from the 101 to the 23 when we had a blowout in the right rear tire. What’s a site visit without a little excitement? We were able to limp off the freeway and transfer the equipment from the Jeep into another vehicle and continue another 10 minutes to the site. Cecil waited for AAA to arrive to put on the spare tire.

Cecil arrived at the site about 30 minutes later. The backup 220 repeater was swapped out with the repaired repeater. The UHF analog repeater was re-channeled and the duplexer re-tuned. After about two hours on the site it was time to head to Johnny’s to pick up pastrami sandwiches and zip down an uncrowded freeway back to San Diego County. The trip was a success and we had our backup 220 repeater ready for Sunset Ridge.

The following week we made another attempt at Sunset Ridge. This time we found the bottom gate locked for the first time. This gate was locked because the Forest Service determined that the paved road was not safe for regular vehicle traffic. However, the weather had been improving and it didn’t appear that we should expect any snow or ice on the pavement. Since the gate was locked we drove back to the same US Forest Service office we had visited before. Because of the COVID-19 lock-down, the office was closed so we returned to the gate to stare at it and consider our next move. Lucky for us there was a combination lock on the gate and it turned out that the same combination that opens the trail-head gate opened the bottom gate. We were once again on our way to the hilltop.

Sunset Ridge is a wild ride. Much more so than Toro Peak because the road is not nearly as well maintained. Even with a Trail Boss Jeep, the road is challenging. This time the road was suffering from snow melt runoff damage and the ruts were deep and frequent. After a bone jarring kidney crushing hour long trek, we made it to the site. There we found that the repeater was operational but that the external power amplifier had failed. I removed the amplifier from the cabinet and reconfigured the repeater for operation without the PA. The site was full of expired rodents and active spiders. We did some cleanup and packed up for the equally harrowing trip back down the hill.

The very next day we were back on the road to Toro Peak. By this time the troubles at the site had accumulated and, in addition to the problem with the analog repeater, we no longer had our microwave Internet connection. We picked up a new gate key in the morning at the Reservation office and headed up the trail to the top. As before, we were prepared with a backup repeater and a replacement power amplifier.

We arrived at the site prepared for as many problems as we could imagine based on the data we were able to collect from PAPA members on the ground in the Coachella Valley. We brought a new microwave microwave radio, new outdoor CAT5 cable, a replacement analog repeater, and a new 100W power amplifier. We also had a tower climber so that the microwave radio could be replaced as it?s located on the rear of the tower mounted dish.
The first priority was restoration of the Internet connection to the site. The data connection to the microwave radio was exercised with no improvement. Ira was able to remotely connect to the microwave radio through the network but unable to see the router beyond that. Next, we put a new connector on the tower end of the cable and tried again. No joy. A further inspection revealed that the replacement connector was not properly terminated and it was redone. Again, no joy.

Next, the old microwave radio was replaced with a new one. The new radio was installed and the connection to the router was still not realized. Next, we ran a new CAT5 cable from the tower to the router inside the cabinet in the shelter. That didn?t work either.

I brought the old microwave radio into the shelter and set it up on the bench. We tried a number of different POE devices and there was still no connectivity between the microwave radio and the router. As part of the testing I plugged the old microwave radio into a different port on the router and the link light immediately lit up! I tried another port on the router with the same success. Then I went back to the designated WAN port on the router and nothing worked. It appeared that the WAN port on the router was dead but the other ports were still serviceable. This discovery was a major breakthrough in our quest to restore Internet connectivity to the site. Ira was finally able to see the router.

In an effort to keep moving forward and not waste time, a new CAT5 cable was run from the tower into the shelter and along the cable trays to the PAPA cabinet. There was no additional troubleshooting. It?s possible that the old microwave radio is actually operational and the problem was just a dead router port.

There was a commercial two-way radio company in the building with us working on their equipment that had also lost Internet connectivity at the same time back in November. They ended up having to replace all of their data infrastructure in the building and attributed the failure to a lighting storm. Their hardware failed at the same time the PAPA hardware failed.

Since they arrived at the site earlier than we did, they were already up and running and had Internet connectivity restored. They allowed us to connect to their switch so that Ira could connect to the PAPA router using a Team Viewer session on my computer. Ira was able to remotely reconfigure the router and re-designate an operational port as the new WAN port. Internet service was restored!

I connected the service monitor to the antenna port on the top of the cabinet to test the existing RF power amplifier. I measured 0.6W at the antenna port and proceeded to swap out the PA with the new one that we brought. Much to my disappointment, the new PA also measured 0.6W at the antenna port. Next, I connected the cable that goes between the PA and the intermod panel and found that I measured 66W out of the PA. Further testing revealed that no power was being passed through either the single stage isolator or the bandpass cavity that served as the intermod panel.

I bypassed the PA and intermod panel feeding exciter power (7W) directly to the transmit side of the duplexer. The power measured at the antenna port remained below 1W. Next, I tested each of the BpBr cavities in the transmit leg of the duplexer and found that the cavity closest to the intermod panel did not pass any power. I was able to get the cavity to work by operating the tuning plunger. I re-tuned the cavity as best I could using just the service monitor and now I was measuring 6W at the antenna port. A quick on-air test proved that the repeater was operational again. It appeared that the receiver side of the duplexer was working fine.

After the repeater was returned to service and the router reconfigured to provide Internet to the analog and digital clients, it became clear that there was no connectivity to the PAPA network on the analog side. Further investigation revealed that the RTCM (Radio Over IP) device was completely dead. It needed to be replaced.

We left the site on Wednesday with the Internet restored, and new microwave radio installed, a new CAT5 cable run from the microwave radio to the cabinet, the defective intermod panel removed, the PA removed, a duplexer cavity made working, and a reconfigured router. The DMR repeater was operating normally on BrandMeister and the analog repeater was operational but not connected to the network.

It looked as though the lightening storm in November caused numerous failures at the site. On the drive down the hill we discussed a return trip with a replacement router, a new RTCM, and the network analyzer to properly re-tune the duplexer.

Cecil and I set out again two days later for Toro Peak. This time we had a new router and RTCM for the site. The drive up the trail to the top was challenging due to dozens of weekend warrior drivers anxious to get out on a nice day and camp on the mountain. The drive down was even worse as we encountered around 10 trucks coming up the hill. The road is only wide enough for a single vehicle in most places.

Once at the site, we replaced the defective router with the new one. It fired right up and the WAN was returned to its original port. Next, the RTCM was replaced and aligned with the controller resulting in restoration of the analog repeater to network service. Using my scalar network analyzer, I was able to realign the cavity that I adjusted two days earlier to the exact operating characteristics required of the duplexer. The last action at the site was to align the analog input and output PL tones to the regional plan. After about 5 hours on the site, we left for the three hour drive back to San Marcos.

Due to the elevation of the site (approx 9000ft.) everything takes longer than you would expect and it?s easy to become fatigued with even the simplest of tasks. Some cleanup work and minor rewiring remains to be done at the site. This work will be done during a future site visit later this summer.

Cecil and I weren’t the only ones burning up the hilltop trails. During this time, Steve, WD8CIK, carried on maintenance and upgrade activities on a number of PAPA sites. Two trips to Santiago Peak (P03) resulted in the installation of the new repeater at the site with RF performance better than it has been in years. The same is true for Mt. Wilson (P08) where a new repeater was also installed. Additionally, Oat Mountain (P01) and Blueridge (P14) both received new repeaters. These upgrade efforts are aimed at providing improved coverage, increasing system reliability, and developing hardware uniformity across the networked sites.

We’re not done yet. There are more sites in the network slated for upgrades over the next few months. In addition to the hardware upgrades, we are implementing our system wide PL plan in order to allow PAPA to effectively co-channel as many repeaters as we can to improve service without requiring additional spectrum resources.

Many people contributed to the efforts described in this article, not just the few making the actual trips to the mountaintop sites. The PAPASYS TECHTEAM is always busy behind the scenes. PAPA relies on an all volunteer technical team to accomplish this amazing work. Multiple disciplines are required from RF systems, network systems, mechanical systems, and documentation to keep everything running and available to the membership. The work requires extensive planning, execution and time.

Building on the success that PAPA has had with Motorola DMR repeaters over the past few years, we are undertaking a system wide upgrade of the 30 year old analog repeaters to newer Motorola repeaters. The legacy repeaters have required extensive maintenance recently as critical components age, fail, and become more difficult to source. The new Motorola repeaters are continuous duty high-power units in a smaller footprint than the legacy repeaters. At a very attractive price point, the Motorola repeaters allow us to maintain a stock of spares to speed hilltop maintenance when necessary. The current plan is to have all analog repeaters upgraded by the end of 2020.

This important hardware upgrade program is not funded by the regular membership dues of the PAPA membership, but rather, by the generous donations of corporate sponsors and individual PAPA members. Our corporate sponsors provide equipment donations equipment upgrades. When you hear these incredible PAPA members on the air, please thank them for their ongoing support of the System. Corporate sponsors include: Bob Yuen, KJ6UVT of Anytone; Kevin Zanjani, KI6DHQ, of Bioenno Power; Jim McLaughlin, KI6ZUM, of ZUM Spots; and Marvin Munster, W6MJM, of Impluse Electronics.

You don’t need expensive RF test equipment, a tower climbing safety harness, knowledge of computer networking, or deep pockets to help with the repeater site equipment upgrade effort. Without the generous financial support of the PAPA membership, none of this would be possible. PAPA is member supported and this is what enables us to continuously improve the System to provide the best service possible to the membership and the organizations and communities that we serve. Please visit www.papasys.com and click the DONATE button to contribute to help the PAPA System, Southern California’s Premier Repeater System.

73,
AA4CD